Systems and methods for creating virtual objects in a sketch mode in a haptic virtual reality environment

ABSTRACT

A user of a modeling application modifies an initial virtual object using a sketch drawn on one or more construction planes. Typically, construction planes are connected by an axis that intersects the virtual object. The user can draw a sketch on each construction plane, and the modeling application interpolates a shape along the axis between the sketches to determine what material in the virtual object is to be removed from it. In this manner, material may be removed to create a recess or hole in the virtual object or otherwise to slice away material from the object. A user can use two or more axes and construction planes to produce complex shapes from the initial virtual object. A user can also select a portion of a virtual object and mirror the selected portion. Modifications that the user makes in the selected portion are made correspondingly in the mirrored portion.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation-in-part of U.S. patent applicationSer. No. ______ (not yet assigned) filed Jun. 28, 1999, entitled“Systems and Methods for Interacting with Virtual Objects in a HapticVirtual Environment,” and is based on and claims priority to U.S.Provisional Patent Application Ser. No. 60/093,304, filed Jul. 17, 1998,entitled “Method and Apparatus for Sculpting Virtual Objects in a HapticVirtual Reality Environment,” the entire contents of which areincorporated herein by reference

FIELD OF INVENTION

The invention relates generally to a method and apparatus forinteracting with virtual objects in a haptic virtual realityenvironment, and more specifically to a method and apparatus formodifying virtual objects in a haptic virtual reality environment.

BACKGROUND OF INVENTION

Computers have been long used to design objects in a computerizedenvironment. Examples are CAD (computer-aided design) systems that areused in the design of mechanical parts, such as parts used in anassembly process. For example, a designer may use a CAD system to designparts for an automobile, which are then produced and used in anautomobile assembly line. Such CAD systems require significant trainingand understanding in using the computerized design system. CAD systemsare typically difficult to use and lack the freedom and expressivenessof traditional noncomputerized drawing, sketching, and model-makingprocedures.

One traditional noncomputerized approach is the blue foam technique. Inthis approach, a two-dimensional (2-D) cardboard template is made andattached to one side of a block of blue foam material. The designer usesa hot wire to trace out the shape of the template throughout the bluefoam block and then removes the excess foam.

Another noncomputerized approach involves modeling the object from clayusing a traditional sculpting approach based on hand tools and handsculpting techniques to add, remove, and shape the clay.

These approaches suffer from traditional limitations such as the timeneeded to hand design templates, and the difficulty or inability torecover from mistakes or return to a previous stage of design. Objectscreated using traditional physical modeling methods cannot be directlyincorporated or used in modern digital CAD and computer-aidedmanufacturing (CAM) processes. In addition, blue foam and clayapproaches cannot be combined readily in the same model.

A more modern approach uses a virtual reality technique to model objectsin a computerized virtual environment. Virtual Reality (VR) is anartificial environment constructed by a computer which permits the userto interact with that environment as if the user were actually immersedin the environment. Early VR devices permitted the user to seethree-dimensional (3-D) depictions of an artificial environment and tomove within that environment. The reality of the VR environment isenhanced by the ability of a user to manipulate virtual objects withinthe virtual environment using hand motions and gestures. A user may usea virtual tool to manipulate and/or modify a computerized model orvirtual object in the virtual environment.

Virtual reality techniques often do not provide for a realistic feelingof sculpting by the user. The user cannot feel when virtual tools touchor modify the virtual object. Moreover, in some cases, the virtual toolmay pass through the virtual object without any impediment, therebyseverely degrading the realism of the experience and the user's abilityto finely control the modification.

SUMMARY OF INVENTION

Thus, there is a need for a 3-D computerized modeling system thatovercomes the problems of CAD techniques and traditional noncomputerizedmodeling techniques. One object of the present invention is to provide acomputerized 3-D virtual reality modeling system that provides the easeand expressiveness of traditional sketching, drawing, and handmodel-making approaches as well as integration into the digital designand manufacturing process offered by CAD/CAM techniques

Being able to feel the virtual object allows the user to resolve visualambiguities, such as a shape that may appear either concave or convex asperceived by an observer. The user may rely on haptic feedback whenmodifying the object such as scratching a slight groove in the object,which the user then deepens or expands while receiving feedback throughthe tool on the shape and current depth of the groove. Feedback alsoallows the designer to monitor and modulate the cutting rate or amountof change induced by the virtual tool. Haptic feedback also helps theuser navigate around and on the surface of the object; that is, usingthe feel of the object to know where the virtual tool is on the object.

The invention also relates to a method for modifying a virtual object ina haptic virtual environment, including sensing a location of a user inreal space; determining a virtual tool comprising discrete points foruse by the user in the haptic virtual environment; producing a mirroredvirtual object by mirroring an initial portion of the virtual object togenerate two mirrored portions; determining the locations of thediscrete points of the virtual tool relative to a selected one of thetwo mirrored portions of the mirrored virtual object; calculating aninteraction force based on the locations of the discrete points of thevirtual tool and the location of the selected portion of the mirroredvirtual object; and modifying the selected portion of the mirroredvirtual object based on the locations of the discrete points of thevirtual tool and the location of the selected portion of the mirroredvirtual object.

BRIEF DESCRIPTIONS OF THE DRAWINGS

The invention is pointed out with particularity in the appended claims.The above and further advantages of this invention may be betterunderstood by referring to the following description taken inconjunction with the accompanying drawings, in which:

FIG. 1 depicts a functional block diagram of a system for one embodimentof the invention, including a haptic interface device, modelingapplication, and graphics display;

FIG. 2A provides a pictorial view of a virtual environment including avirtual object and a virtual tool, for one embodiment of the invention;

FIG. 2B provides a pictorial view of a virtual tool contacting thevirtual surface of a virtual object in connection with a hapticinterface location within the virtual object, for the embodiment of theinvention shown in FIG. 2A;

FIG. 3 illustrates a high-level flowchart of the haptic renderingprocess between a virtual tool and virtual object, for one embodiment ofthe invention;

FIG. 4 is a high-level flowchart of the haptic rendering process betweena virtual tool and a virtual object for another embodiment of theinvention;

FIG. 5A illustrates a pictorial view of a virtual tool approaching thevirtual surface of a virtual object;

FIG. 5B illustrates a pictorial view of a proposed location for avirtual tool with one point of the virtual tool penetrating the virtualsurface of the virtual object, for the embodiment shown in FIG. 5A;

FIG. 5C illustrates a pictorial view of a proposed location for thevirtual tool with two points of the virtual tool penetrating the virtualsurface of the virtual object, for the embodiment shown in FIG. 5A;

FIG. 5D illustrates a pictorial view of a virtual tool moving along thevirtual surface of a virtual object, for the embodiment shown in FIG.5A;

FIG. 5E illustrates a two dimensional pictorial view of a virtual toolmoving along a concave edge in the virtual surface of a virtual objectfor the embodiment shown in FIG. 5A;

FIG. 5F illustrates a three dimensional pictorial view of a virtualobject moving along a concave edge, for the embodiment shown in FIG. 5E;

FIG. 6 illustrates a virtual tool with points located throughout theinterior of the tool;

FIGS. 7A-7C depict flowcharts of the haptic rendering process between avirtual object and virtual tool for one embodiment of the invention;

FIG. 8A illustrates a pictorial view of a virtual tool encountering theconvex edge of a virtual object for one embodiment of the invention;

FIG. 8B illustrates a pictorial view of a surface direction vectorcalculated for one proposed tool position for the embodiment of theinvention shown in FIG. 8A;

FIG. 8C illustrates a pictorial view of a surface direction vectorcalculated for a second proposed tool position for the embodiment shownin FIG. 8A;

FIG. 8D illustrates a pictorial view of virtual tool constrained to theedge of a virtual object for the embodiment of the invention shown inFIG. 8A;

FIG. 9 depicts a set of voxels with different density values and anisosurface;

FIG. 10 illustrates a two dimensional representation of a rectangularsolid with voxels and an isosurface, for the embodiment shown in FIG. 9;

FIG. 11 illustrates a ramp length diagram comparing voxel values topenetration distances for one embodiment of the invention;

FIG. 12 illustrates a pictorial view of the surface direction vector ofa point and surrounding density evaluation points for one embodiment;

FIG. 13 illustrates a schematic view of a gradient for a virtual object,including a last SCP, a current SCP, an initial point, and a midpointfor one embodiment;

FIGS. 14A, 14B, and 14C illustrates pictorial views of a virtual surfaceand the endpoints of a segment that intersects the a virtual surface ofa virtual object for one embodiment of the invention;

FIG. 15 illustrates a pictorial view of previous surface contact points,tangency planes, and resulting surface contact point for one embodimentof the invention;

FIGS. 16A and 16B show a pictorial view of a spherical virtual tool in achannel formed between two virtual objects and;

FIGS. 17A-17E show pictorial views of a virtual tool encountering asurface and moving along the surface constrained by a constraint plane.

FIG. 18 illustrates a flowchart of the modification process occurringbetween a virtual tool and virtual object, for one embodiment of theinvention;

FIG. 19A provides a pictorial view of a swept area based on a removalline between two points, for one embodiment of the invention;

FIG. 19B provides a pictorial view of endpoints, a desired point, aprojected point, endpoint planes, and a projected plane, for theembodiment of the invention illustrated in FIG. 19A;

FIG. 20 provides a pictorial view of a cube near a virtual surface of avirtual object with a protruding feature of the virtual surfacepenetrating between two points of the cube;

FIG. 21 provides a pictorial view of a sphere penetrating a virtualsurface between two points of the sphere;

FIG. 22 provides a pictorial view of a sphere represented by multiplepoints penetrating a virtual surface, as well as illustrating acalculated distance between the origin of the sphere and the virtualsurface;

FIG. 23A-23D illustrates pictorial views of a virtual tool withoutrigger constructs including arms and outriggers connected to thehandle of the virtual tool;

FIG. 24 illustrates a pictorial view of a painted area of the surface ofa virtual object and a corresponding selection volume;

FIG. 25 illustrates a pictorial view of a virtual tool moving through avirtual object, constrained by the geometric constraint of a curvedline;

FIG. 26 illustrates a flowchart of the process of setting a geometricconstraint and using the geometric constraint to constrain the movementof a virtual tool; and

FIG. 27 illustrates a pictorial view of a virtual lathing tool and avirtual object rotating in virtual space under the control of a controlwheel.

FIG. 28A illustrates a pictorial view of an initial block of virtualmaterial overlaid by construction planes aligned along an axis and asketch on the top surface of one construction plane for one embodimentof the invention;

FIG. 28B illustrates a pictorial view of a resulting block of materialbased on the sketch for the embodiment of the invention depicted in FIG.28A;

FIG. 28C illustrates a pictorial view of another approach to obtaining aresulting block of material based on the sketch for the embodiment ofthe invention depicted in FIG. 28A.

FIG. 29A illustrates a pictorial view of the cut block of FIG. 28Boverlaid with a potential material of the virtual environment, alongwith a starting construction plane, ending construction plane, and asketch on the starting plane;

FIG. 29B illustrates a pictorial view of a union block of material forthe embodiment of the invention depicted in FIG. 29A;

FIG. 30 illustrates a flowchart of the process of producing a modifiedvirtual object from an initial virtual object without requiring anyforce feedback interaction for one embodiment of the invention;

FIG. 31 illustrates a flowchart showing the process of a producing amodified virtual object from an initial virtual object in a hapticvirtual environment while providing force feedback to the user for oneembodiment of the invention;

FIG. 32A illustrates a pictorial view of an initial virtual object withsketches on several construction planes for one embodiment of theinvention;

FIG. 32B illustrates a pictorial view of modified virtual object createdby an interpolation process between the sketches shown in the embodimentfor FIG. 32A;

FIG. 33A illustrates a pictorial view of a virtual object withconstruction planes connected by an axis for one embodiment of theinvention;

FIG. 33B illustrates a pictorial view of the axis that intersects thevirtual object at a different angle for the embodiment show in FIG. 33A;

FIG. 34A illustrates a pictorial view of virtual object with a pointconstraint on a sketch on a construction plane for one embodiment of theinvention;

FIG. 34B illustrates a pictorial view of a virtual object with a sketchplane showing a parallel line constraint for one embodiment;

FIG. 34C illustrates a pictorial view of virtual object with a sketchplane showing a perpendicular line constraint for one embodiment;

FIG. 34D illustrates a pictorial view of virtual object with a sketchplane showing a circle constraint for one embodiment;

FIG. 35 illustrates a pictorial view of snap-to area for a point, forone embodiment of the invention;

FIG. 36 illustrates a block diagram of a scene graph, including a sketchblock node, plane stack nodes, and sketch plane nodes;

FIG. 37 illustrates a pictorial view of an open curve to be trimmed;

FIG. 38 illustrates a pictorial representation of a sketch plane showinga grid with a closed outline, vertical grid lines and horizontal gridlines;

FIG. 39 illustrates a pictorial representation showing the interpolationprocess between a point V₀ on a sketch plane and a point V₁ on a sketchplane resulting in a new point V_(N), on an intermediate plane;

DETAILED DESCRIPTION OF THE INVENTION

The description includes headings and subheadings that aid in organizingthe text, but are not meant to be limiting in any way. Topics discussedunder a heading or subheading may also be described elsewhere throughoutthe specification.

FIG. 1 shows a system for one embodiment of the invention, including ahaptic interface device 10, modeling application 12, and graphicsdisplay 14 in electrical communication with each other. The modelingapplication 12 includes a haptic rendering process 16, a virtual tooland object interaction process 18, a virtual object modification process20, a graphics process 22, and a 3-D printer 23 all in electricalcommunication with each other.

Generally, a user of the system uses the haptic interface device 10 tointeract with the virtual object 26 (see FIG. 2) receiving forcefeedback produced by the haptic rendering process 16 and viewinggraphics rendered by the graphics process 22 on a graphic display 14.

The user may also output the virtual object 26, or a surfacerepresentation of it, to a 3-D printer 23 to produce an actual 3-Dphysical model of the virtual object 26. In one embodiment, the 3-Dprinter is a stereolithography machine.

Software Process

In one embodiment, a “process”, as referred to in FIG. 1, is a softwareprocess executing on a hardware microprocessor. All the processes mayexecute on one microprocessor, or in other embodiments, one or moreprocesses may execute on different microprocessors, which are linked bybuses, cables, local networks, wide area networks, or global computernetworks, such as the Internet.

The modeling application 12 is viewed in FIG. 1 as a softwareapplication executing on one computer system. In another embodiment, themodeling application 12 executes on one or more computer systemsconnected by a communications device, such as a bus, cable, or networkconnection. In an alternate embodiment, the modeling application is ahardware device, such as an ASIC (application specific integratedcircuit), and one or more processes of the application are implementedon one or more ASIC devices. In a further embodiment, the modelingapplication 12 is implemented as one or more objects, which may executeon one or more computer systems. In one embodiment, the modelingapplication 12 runs on a dual 300 MHz Intel® Pentium® 2 computer runningMicrosoft® Windows NT™ 4.0 using an Open GL accelerated graphics card.

The modeling application 12 is not required to include a hapticrendering process 16, an interaction process 18, a modification process20, and a graphics process 22. In one embodiment, the functions of themodeling application 12 are implemented by a different number ofprocesses. In one embodiment, the modeling application 12 includes thehaptic rendering process 16 and the graphics process 22.

In one embodiment, the invention is implemented using an object-orientedapproach. The haptic rendering process 16 and other processes areimplemented as software objects. In another embodiment, the virtualobject 26 and the virtual tool 28 are implemented as software objectsand perform one or more of the functions of the haptic rendering process16. In one embodiment, the virtual tool 28 is a software object thatperforms such functions as determining if contact has occurred with avirtual object 26 and determining the surface direction vector 101, aswill be discussed later.

In one embodiment, the virtual object 26 and the virtual toot 28 areimplemented as software objects in the C++ programming language. Inother embodiments, the virtual object 26 and virtual tool 28 areimplemented using an object-oriented programming language other thanC++.

In one embodiment, the modeling application is a computer program storedon a computer readable storage media, such as a CD disc, diskette, tape,or other media. In another embodiment, the modeling application is acomputer program distributed over a computer-readable propagated signal,such as a program distributed over the Internet.

Haptic Interface Device

In one embodiment, the system includes a haptic interface system, asshown in FIG. 1, including the haptic interface device 10 and the hapticrendering process 16 which generates a virtual object of the virtualenvironment to be “touched” and determines the results of theinteraction (discussed in more detail below). The haptic interfacedevice 10 is a tactile or force-feedback device which provides the touchsensations of interacting with virtual objects 26 to a user of thesystem. Some haptic interface devices 10 consist of an electromechanicallinkage which can exert a controllable force on a user's hand. See, forexample, U.S. Pat. No. 5,625,576 issued to Thomas H. Massie and J.Kenneth Salisbury, Jr., the disclosure of which is herein incorporatedby reference in its entirety. As used herein, “haptic rendering” refersto the creation of a virtual environment with which a user can interactthrough the sense of touch. The term “haptic rendering process” 16refers to the computer program which generates the haptic aspects of thevirtual environment and determines the forces to be applied to a userthrough a haptic interface. The haptic rendering process 16 generateshaptic representations of virtual objects in the virtual environment.

Overview Of Device, Virtual Object And User Interaction

FIG. 2A shows a haptic virtual environment including a virtual object 26and a virtual tool 28. The virtual object 26 of the embodiment shown inFIG. 1 is depicted as a 3-D (three dimensional) block of materialtypically “floating” in the virtual space of the virtual environment.The virtual object 26 has a virtual surface 25 that represents the“skin” of the virtual object 26. The virtual tool 28 is represented inFIG. 2A as a sphere 34 with a rod or “handle” 32 connected to it.

In one embodiment, the user uses a haptic interface device 10 in realspace to grasp or manipulate the handle 32 of the virtual tool 28 invirtual space. In one embodiment, the location of this handle withrespect to the virtual tool 28 can be changed interactively by the user.As used herein, a “haptic virtual environment” refers to acomputer-generated virtual environment that can be explored by a userthrough the sense of touch. In one embodiment, the haptic virtualenvironment contains a virtual object 26 that is model of a real worldobject that a user is creating in the virtual environment. In anotherembodiment, the haptic virtual environment incorporates two or morevirtual objects 26 that are linked to each other, such as in ahierarchical arrangement. It should be understood that the interactionand/or modification methods described herein may be readily extended toapply to two or more virtual objects 26 linked or associated in a hapticvirtual environment.

FIG. 2B illustrates a virtual tool 28 contacting the virtual surface 25of a virtual object 26. The user guides the virtual tool 28 using thehaptic interface device 10, represented, in this embodiment, by a stylus33 in FIG. 2B. The position and orientation of the tip of the stylus 33indicate the haptic interface location 98. Note that, although the usermay be manipulating a literal stylus in some embodiments, the hapticinterface location 98 could be controlled by a user interacting with anynumber of differently shaped elements such as a thimble, a yoke, or aball. The tip of the virtual stylus 33 is indicated by the hapticinterface location 98. In one embodiment, the haptic rendering process16 tracks the haptic interface location 98, but does not otherwise trackthe shape or location of the entire haptic interface device 10.

The haptic rendering process 16 attempts to move the virtual tool 28 sothat the origin 27 of the virtual tool 28 matches the haptic interfacelocation 98. However, unless the haptic rendering process 16 is usingthe virtual tool 28 to remove material from the virtual object 26, thenthe haptic rendering process 16 typically does not allow the virtualtool 28 to penetrate the virtual object 26. Thus, as shown in FIG. 2B,the user has attempted to move the virtual tool 28 into the virtualobject 26, which is indicated by the haptic interface location 98 withinthe virtual object 26. The haptic rendering process 16 calculates aresistance to the movement of the virtual tool 28 into the virtualobject 26. This calculation is based on a connection 29 between the toolorigin 27 and the haptic interface location 98, as will be discussed inmore detail later. In one embodiment, the connection 29 includes avirtual spring 31. In one embodiment, the connection 29 includes avirtual dash-pot. Thus, if the user attempts to move the virtual tool 28further into the virtual object 26, the haptic rendering process 16calculates an increasing resistance force that is fed back to the userthrough the haptic interface device 10 based on the virtual spring 31.

In one embodiment, the user is allowed to move the virtual tool 28through the virtual object 26 without resistance while removingmaterial. In this case, the user selects a transparent or translucentmode, and the virtual tool 28 appears translucent. The haptic renderingprocess 16 allows the user to move the virtual tool 28 through thevirtual object 26 without constraint or resistance.

Description of Virtual Tool and Modification Options

As already described, the user interacts with the virtual object 26 inthe virtual environment through a virtual tool 28. The user may selectany shape for the tool 28. The shape of the tool 28, along with othercharacteristics, such as interaction mode, determine the interactionwith the virtual object 26. In one embodiment, the tool 28 may berepresented as a series of discrete points in virtual space whichoutline a three-dimensional shape of the tool 28. The virtual tool 28 ismodeled as a set of discrete points for the purposes of hapticinteraction and collision detection with the virtual object 26. Inanother embodiment, the points of the virtual tool 28 are created by analgebraic equation or any other continuous or piecewise mathematicalmethod suitable for determining a 3-D shape in a virtual environment. Inanother embodiment, the tool 28 can be represented directly bycontinuous or piecewise mathematical equations, rather than by discretepoints. The virtual tool 28 may take on any of a number of shapes thatmay be useful for a user when using a virtual tool 28 to create avirtual object 26 in the virtual environment. Typical shapes may includea sphere or cylinder. In another embodiment, the user selects one ormore interaction modes for the virtual tool 28, such as a sandpapermode, which causes the tool 28 to remove material gradually from thevirtual object 26, much like using real sandpaper to smooth the shape ofa block of wood in the real world.

Creating and Manipulating the Virtual Object

FIG. 3 illustrates a flowchart of the haptic rendering process between avirtual tool 28 and virtual object 26. First, a virtual object 26 isgenerated (step 40). Typically, this occurs when the user requests thatan initial virtual object 26 be created, for example, by directing thatthe virtual object 26 assume a 3-D cube or “block” shape. In oneembodiment, the initial virtual object can be defined as a shapegenerated from a saved file, scanner, or 3D digitizer. In oneembodiment, the user selects an interaction mode that selects thecharacteristic of the virtual object 26. The shape and material, andsurface properties of the virtual object 26 can be specified, forexample, the hardness or softness of the object 26. For example, if theuser selects a sculpting mode, then the virtual object 26 assumescharacteristics generally representative of a block of clay. If the userselects a 3-D sketch mode, then the virtual object 26 is overlaid with astack of planes or slices, and the user sketches a template on thesurface of one of the planes. However, the virtual object 26 is notrequired to assume a cube or block shape and may assume any 3-D shapethat the user finds useful when starting to design a model from thevirtual object 26. In one embodiment, the object properties permit onlymanipulation or movement of the virtual object 26 without any sculptingor other modification thereof. The modification mode is not limited towhat is described here, but, in other embodiments, is based on othermodes of interaction or modification that a user finds useful. Forexample, other modes may include smoothing the surface of the virtualobject 26, shifting material, or mirroring all or parts of the object26.

In one embodiment, the virtual object 26 is created by the modificationprocess 20 under the directions of the user, and then the graphicsprocess 22 displays a corresponding representation of the virtual object26 to the user.

If the user selects a 3-D sketch mode, then the virtual object 26 isoverlaid with a stack of planes or slices, and the user sketches atemplate on the surface of one of the planes.

Generating the Virtual Tool

In the next step, a virtual tool 28 is generated (step 42). In oneembodiment, the user selects a shape and characteristics for the virtualtool 28, which the haptic rendering process 16 generates in the virtualenvironment. The graphic process 22 then displays a correspondingversion of the virtual tool 28 to the user on a graphics display 14. Forexample, the virtual tool 28 can assume different shapes and interactionor modification characteristics. The tool 28 can assume any 3-D shape,such as a sphere 34 or a cube, or a substantially 2-D shape, such as aspatula or knife. In general, modes include a material removal, materialaddition, or other material modification mode such as smoothing. In oneembodiment, removal modes include Boolean removal, sand paper removal,and removal using the concept of a function which falls off gradually.Additional modes include Boolean addition, Boolean pull, and pull usinga full-off function. An additional form of interaction and modificationincludes a sketch mode for sketching a template on the surface of avirtual object 26. Used here, Boolean refers to adding or subtractingtwo geometries to arrive at a third geometry.

Sensing User Location

In the next step, sensors determine the location of a user in real space(step 44). In one embodiment, the sensors are any type of sensors usefulin determining the location of a user, such as the location of a hand inreal space. Such sensors could be based on measurements of locationbased on mechanical, electrical, magnetic, optical, or other sensingdevices. In one embodiment, the haptic interface device 10 senses thelocation of the user in real space. For example, the user physicallymanipulates the haptic interface device 10, such as a handgrip orstylus, in real space and the location of this device is determined inreal space. In one embodiment, one such haptic interface device 10 isthe PHANToM® device from SensAble Technologies, Inc., Cambridge, Mass.Generally, the PHANToM® device can sense six degrees of freedom—x, y, z,pitch, roll, yaw, while providing for force feedback in three degrees offreedom—x, y, z. One embodiment of this invention includes a hapticinterface that can provide more than three degrees of force feedback.

As used herein, “real world space” is defined as the real worldenvironment. The haptic rendering process 16 utilizes the informationobtained by the sensors to determine the haptic interface location 98 inthe virtual environment. As used herein, “haptic virtual environment”refers to the region in the computer generated virtual environment withwhich the user can interact through the sense of touch. The location ofthe haptic interface describes the location of the user in the hapticvirtual environment.

Correlating User Location and Virtual Tool Position

In one embodiment, the haptic rendering process 16 then translates thelocation of the haptic interface device 10 in real space into acorresponding location in the haptic virtual environment, which is thehaptic interface location 98 (step 46). Then the haptic renderingprocess 16 uses a method to limit the movement of the tool 28 based onthe virtual surface 25, the position of the tool 28, and the hapticinterface location 98 (step 54). The objective of the method is to movethe tool 28 as close as possible to the haptic interface location 98,without crossing the virtual surface 25 and using a path for the tool 28that yields progressively better locations for the tool 28, whichtypically means locations closer to the haptic interface location 98.This method is discussed in detail later.

Typically, the haptic rendering process 16 analyzes the interaction andcalculates results based on the potential position of the virtual tool28 relative to the virtual object 26 (step 56). Then the results areapplied to the virtual tool 28 and/or the user (step 58).

Calculation of Force

For example, if the movement of the virtual tool 28 is constrained suchthat the virtual tool origin 27 is not coincident with the hapticinterface, the haptic rendering process 16 may calculate an interactionforce to be applied to the haptic interface device 10, so that the userfeels a resistance to trying to move the virtual tool 28 into thevirtual object 26. In this case, the results are a feedback forceapplied to the user via the haptic interface device 10 and correspondingconstraints or limits on the movement of the virtual tool 28 (step 56).The force feedback provides the user important non-visual informationabout the shape of the virtual object, whether the interaction mode isobject modification or simply evaluating the shape of the object.

Interaction and Modification

In one embodiment, if the user is using the virtual tool 28 in aninteraction mode, such as sandpaper, then the modification process 20calculates changes in the virtual object 26, such as material beingremoved, which in turn changes the graphical representation of thevirtual object 26. The results, in this case, are a modification to thevirtual object 26 (step 58). Interaction modes need not result inmodification of the virtual surface. For example, in another case, theuser may be trying to use the virtual tool 28 to evaluate the shape ofthe virtual object 26 without trying to modify the object 26. In thiscase, the results are limits on the movement of the virtual tool 28without any penetration or modification of the virtual object 26.Another example is if the user presses the virtual tool 28 into avirtual object 26 in an erase or removal mode, but does not press withenough force, then the virtual tool 28 remains at the surface or mayskirt along the surface of the virtual object 26 without removing anymaterial. The results, in this case, are constraints on the movement ofthe virtual tool 28 (step 58). The calculation and application ofresults (steps 56 and 58) are not confined to what is described here butinvolve other effects depending on many factors, such as the shape ofthe virtual tool 28, the characteristics of the tool 28, thecharacteristics of the virtual object 26, the nature of the movement ofthe virtual tool 28 relative to the virtual object 26, and otherfactors. Another factor may be a construction constraint that aids inthe construction of a virtual object 26. In one embodiment, theconstraint can be a line, an arbitrary curve, or a surface thatconstrains the movement of the virtual tool 28.

Ongoing Interaction Between User and System

Finally, after the results have been applied (step 58), the user engagesin additional movement of the haptic interface device 10, in which casea new location must be sensed (step 44) and steps 46-58 are repeated.Alternatively, the user changes the nature of the virtual object 26and/or the interaction mode. The user may also change the shape orcharacteristics of the virtual tool 28 (not shown in FIG. 3). Thesechanges by the user in turn would affect the calculations, constraints,and results determined by steps 46, 54, 56, and 58.

Virtual Object and Tool Interaction

Haptic Rendering Process

FIG. 4 is a flowchart of the haptic rendering process 16 for theinteraction between a virtual object 26 and a virtual tool 28 for oneembodiment of the invention. In the first steps, the haptic renderingprocess 16 generates a virtual object 26 (step 60). The haptic renderingprocess 16 determines or generates a virtual tool 28 represented using aplurality of discrete points for use by the user (step 62). Sensorssense the location of the user in space (step 64) in a manner similar tothat described for FIG. 3 above. The haptic rendering process 16 thendetermines a haptic interface location 98 (see FIG. 5A) for the hapticinterface (step 66) in the haptic virtual environment corresponding tothe location of the haptic interface device 10, which the user ismanipulating in real space. The haptic rendering process 16 determinespotential locations for the points of the virtual tool 28 in the hapticvirtual environment in comparison to the haptic interface location 98and the virtual surface of the virtual object 26 (step 68).

The haptic rendering process 16 determines the amount of penetrationinto the virtual object 26 for all the points of the virtual tool 28 ifit were to be moved to the potential location (step 70). The hapticrendering process 16 may determine that there would be no penetration,that only one point of the virtual tool 28 would penetrate the virtualobject 26, or that several points of the virtual tool 28 would penetratethe virtual object 26.

If at least one of the points of the virtual tool 28 has penetrated thevirtual object 26, then the haptic rendering process 16 determines ageometry for the virtual surface at the area of penetration of thevirtual tool 28 (step 72). For example, the haptic rendering process 16determines if the virtual object 26 has an edge, trough, valley, vertex,or hole in the vicinity of the virtual tool 28, to be discussed in moredetail later with respect to FIGS. 7A-7C. This determination is thenused in the next step (step 74), which determines limits or constraintsfor the movement of the virtual tool 28 based on the geometry of thevirtual object 26 (as determined in step 72), the locations of thepoints of the virtual tool 28 that would have penetrated the virtualobject 26 (as determined in steps 68 and 70), and the haptic interfacelocation 98 (as determined in step 66). The haptic rendering process 16then uses the previously determined constraints to constrain or limitthe movement of the virtual tool 28 (step 76). For example, if thevirtual tool 28 has encountered an edge or trough on the surface of thevirtual object 26, then the virtual tool 28 may be constrained to slidealong the edge or trough (see FIG. 5E) until the user makes a definitivemovement, such as moving away from the virtual object 26. In oneembodiment, the movement of the virtual tool 28 is limited without theuser feeling any force feedback. In another embodiment, the movement ofthe tool 28 is limited and the user feels an interactive force feedbackcorresponding to the limits on movement of the tool 28.

Relationship of Virtual Tool Location to Haptic Interface Location

FIG. 5A illustrates a virtual tool 28 approaching the virtual surface 25of a virtual object 26. The surface of the virtual tool 28 is defined bya series of points, S1, S2, S3, S4, S5, S6 in a simplified view of avirtual tool 28 in one embodiment.

In another embodiment of the invention, the points that define thevolume of the virtual tool 28 extend throughout the space of the virtualtool 28 (shown as additional interior points 112 within the tool 28 inFIG. 6). In one embodiment, the additional points 112 are spaced evenlythroughout the interior 3-D volume of the tool 28. In anotherembodiment, there is no requirement that the additional points 112 bespaced evenly within the tool 28.

In the example shown in FIG. 5A, the user is moving the haptic interfacedevice 10 so that the virtual tool 28 is moving toward the virtualsurface 25 of the virtual object 26. The haptic rendering process 16attempts to move the origin 27 of the virtual tool 28 to match thehaptic interface location 98. In the embodiment shown in FIG. 5A, thelocation of the virtual tool origin 27 lags behind the haptic interfacelocation 98 as the user moves the virtual tool 28 through the virtualspace.

Haptic Rendering and Graphics Rendering

It should be understood that the haptic rendering process 16 isoperating at a high rate of speed, such as updating the location of thehaptic interface location 98 and virtual tool 28 many times per second,as the user attempts to move the virtual tool 28 through the virtualenvironment. In one embodiment, the haptic rendering process 16 isupdating the locations at about 1000 times per second. In oneembodiment, some calculations of the haptic rendering process, such asforce reaction calculations, are occurring at about 1000 times persecond, while less time critical calculations of the haptic renderingprocess, such as geometry calculations, are occurring at slower rates,such as 100 times per second. The graphics process 22 updates the imageseen by the user on the graphics display 14, but typically at a lowerrefresh rate than the haptic rendering process 16. In one embodiment,the graphics process 22 updates the graphics display 14 at a rate ofabout 60 times per second.

Example of Tool Points Penetrating Virtual Object

FIG. 5B illustrates one point S4 of a virtual tool 28 encountering thevirtual surface 25 of the virtual object 26, for the embodiment shown inFIG. 5A. The haptic rendering process 16 attempts to move the virtualtool origin 27 to match the location of the haptic interface 98, butdetermines that a point, S4, of the virtual tool 28 would cross thevirtual surface 25. The haptic rendering process 16 then determines theapproximate minimum distance vector 101 in a direction toward thevirtual surface for a point S4. This vector 101, later referred to asthe surface direction vector 101, also has the property that it is theapproximate normal vector to the virtual surface 25 where it crosses thesurface 25. As described in more detail later, the haptic renderingprocess 16 uses this vector 101 to calculate movement constraints forthe virtual tool 28, as the virtual tool origin 27 is progressivelymoved toward the haptic interface location 98. In general, the hapticrendering process 16 attempts to keep the points of the tool 28 outsideof the virtual surface 25 of the virtual object 26; that is, the object26 is treated as a solid.

Example of More than One Point Encountering the Virtual Object

FIG. 5C illustrates two points, S3 and S4, of the virtual tool 28encountering the virtual surface 25 of a virtual object 26, for theembodiment shown in FIG. 5A. In this case, the virtual tool 28encounters the virtual surface 25 at a different orientation than theone shown in FIG. 5B. For example, the user has rotated the virtual tool28 slightly or is moving the virtual tool 28 at a slightly differentangle, so that two points, S3 and S4, are encountering the virtualsurface 25 rather than the one point, S4, as shown in FIG. 5B. Thesurface direction vector 101 shown in FIG. 5B is based on point S4. InFIG. 5C, the surface direction vector 101 is based on point S4 as it haspenetrated the virtual surface 25 more than point S3. In the embodimentshown, if multiple points penetrate the virtual surface 25, the surfacedirection vector 101 is calculated for the point of deepest penetration.Other embodiments may use a combination of all points which arepenetrating to calculate a resulting minimum surface direction vector101.

Local Geometry Constrains Motion

FIG. 5D illustrates a virtual tool 28 skirting along the virtual surface25 of a virtual object 26, for the embodiment shown in FIG. 5A. A planeof tangency or constraint plane 104 is formed that is orthogonal to thesurface direction vector 101 that was previously calculated. In FIG. 5D,the plane of tangency 104 is shown aligned with the origin 27 of thevirtual tool 28. In other embodiments, the plane of tangency 104 may bealigned differently, such as at or near the virtual surface 25 of thevirtual object 26, as long as the plane of tangency 104 constrains themovement of the virtual tool 28 substantially parallel to the virtualsurface 25. In FIG. 5D, the plane of tangency extends out of the planeof the diagram. The virtual tool 28 is allowed to move along the planeof tangency 104. The movement is shown by the direction of movement ormotion vector 106 a.

FIG. 5E provides a two dimensional view of a virtual tool 28 constrainedto move along an inside or concave edge 108. FIG. 5F illustrates a 1three dimensional view of the virtual tool 28 constrained to move alongan edge 108 of a trough or valley 110 for the embodiment of theinvention shown in FIG. 5E. The virtual tool 28 is constrained to movein the direction of a vector 106 b that constrains the tool 28 to movein a direction approximately parallel to the concave edge 108 toward ahaptic interface location 98. FIG. 5F provides another pictorial view ofa the tool 28 moving along a vector 106 b approximately parallel to theconcave edge 108 of the virtual object 26.

Modification of the Virtual Object

If the virtual tool 28 is in a removal mode, the modification process 20may also determine material to be removed from the virtual object 26 inresponse to the shape of the virtual tool 28. In one embodiment, thisremoval mode is termed a carving mode. In one embodiment, the user feelsa resistive force when attempting to move the virtual tool 28 into thevirtual object 26 because forward motion of the virtual tool 28 into theobject 26 is not allowed. Forward motion of the tool 28 is only achievedas a result of the modification process 20 changing or deforming thegeometry of the virtual object 26 in a manner that simulates carving ofthe virtual object 26. The haptic rendering process 16 continues totreat the virtual object 26 as a solid object as material is removed oradded. For example, the user feels added material as solid, sincemovement of the virtual tool 28 into the material is not permitted.

Case 1 of Tool in Empty Space

FIGS. 7A-7C depicts flowcharts of the haptic rendering process 16 whichcalculates interactions between the virtual object 26 and the virtualtool 28 for one embodiment of the invention. These flowcharts representmovement of the virtual tool 28 in steps relative to the virtual surface25 of a virtual object 26. First, the haptic rendering process 16proposes to move the virtual tool origin 27 toward the haptic interfacelocation 98 (step 120). Then the haptic rendering process 16 determinesif any of the tool 28 points would penetrate the virtual surface 25(step 122). If none of the tool 28 points would penetrate the virtualsurface 25, this represents “Case 1.” The virtual tool 28 wouldencounter only empty space by moving to the proposed new location, sothe incremental movement of the virtual tool origin 27 is allowed (step121) and the haptic rendering process 16 returns to step 120.

Case 2 of Tool Penetrating the Virtual Surface

If at the proposed tool position, some points of the virtual tool 28would penetrate the virtual surface 25 (step 122), then the hapticrendering process 16 finds the direction of shortest distance to thevirtual surface as indicated by surface direction vector 101 for thepoint of the greatest potential penetration into the virtual object 26by the virtual tool 28 (step 126, see also FIGS. 5B and 5C). The hapticrendering process 16 then calculates a constraint plane or plane oftangency 104 (see FIG. 5D) based on the surface direction vector 101(step 127 of FIG. 7A). The plane of tangency 104 is a plane orthogonalto the surface direction vector 101. The haptic rendering process 16then attempts to move the virtual tool origin 27 toward the hapticinterface location 98 but constrains the movement to the plane oftangency 104 (step 130) to arrive at a second proposed virtual tool 28location.

If none of the tool 28 points would penetrate the virtual surface 25 atthis second proposed virtual tool position (step 132), then the hapticrendering process 16 moves the virtual tool origin 27 to the secondproposed location (step 121) and returns to step 120. This situationrepresents “Case 2” and the haptic rendering process 16 has assumed thatthe virtual tool 28 is intended to be touching the surface 25 of thevirtual object 26.

Case 3 of Tool Encountering Edge Condition

If at the second proposed tool position, some points of the virtual tool28 would penetrate the virtual surface 25 (step 132), then the hapticrendering process 16 finds the two surface direction vectors 181 a and181 b of the virtual object 26 at the tool's 28 two points of greatestpenetration at each of the previous two proposed tool positions (step136). FIG. 8A illustrates a virtual tool 28 encountering the outside orconvex edge 177 of a virtual object 26 in one embodiment of theinvention. The virtual object 26 extends out of the plane of thediagram, and has an edge 177 that likewise extends out of the plane ofthe diagram. In FIG. 8B the proposed tool position penetrates thevirtual object 26 at point S4 ₁ and the haptic rendering process 16calculates a surface direction vector 181 a relative to virtual surface25 a. In FIG. 8C, the haptic rendering process 16 proposes a secondproposed tool location based on the surface direction vector 181 arelative to virtual surface 25 a, as shown in FIG. 8C. In this proposedtool location, point S3 ₂ has penetrated the virtual object 26, and thehaptic rendering process 16 calculates the surface direction vector 181b relative to virtual surface 25 b.

The haptic rendering process 16 uses the vectors 181 a and 181 b and thedesired direction of movement to determine constraints to the toolmotion, as follows. The haptic rendering process 16 calculates the crossproduct of the vectors 181 a, 181 b found in step 136 to generate anedge line and a direction of movement 182 (step 138 in FIG. 7B), asshown in FIG. 8D. The haptic rendering process 16 then determines thedot product of each minimum distance vector 181 a, 181 b with thedirection of movement 182, as indicated by the user (step 140). Thehaptic rendering process 16 then determines if either dot product isgreater than zero (step 140). If one of the dot products is greater thanzero, then the haptic rendering process 16 takes the average of bothsurface direction vectors 181 a, 181 b (step 142). The haptic renderingprocess 16 then proposes to move the tool 28 in the direction indicatedby the average vector (step 144) and then proceeds to Case 2 (step 146).The haptic rendering process 16 then proceeds to step 127 in FIG. 7A todetermine a plane of tangency 104 using the average normal.

If neither dot product is greater than zero (step 140), then the hapticrendering process 16 constrains movement to the previously determinededge line (step 148). See FIG. 8D, which shows the virtual tool 28 atthe edge 177 of the virtual object 26. In one embodiment, the virtualtool 28 is constrained based on one point of the tool 28, such as theorigin 27 of the tool 28. The haptic rendering process 16 then attemptsto move the virtual tool origin 27 toward the haptic interface location98 but constrains the movement to the direction of the edge 108 or 177(step 148). If none of the tool points would penetrate the virtualsurface 25 at this third proposed virtual tool position (checked in step149), then the haptic rendering process 16 moves the virtual tool origin27 to this proposed location (step 121) and returns to step 120.

This situation represents “Case 3,” and the haptic rendering process 16assumes that the virtual tool 28 is on an edge 108 or 177 of the virtualobject 26. Generally, in Case 3, the haptic rendering process 16attempts to properly identify an edge 108 or 177 of the virtual object26 and allow movement of the virtual tool 28 along the edge 108 or 177.

In one embodiment the haptic rendering process 16 identifies an edge 108or boundary between two portions of the virtual surface 25 of a virtualobject 26. The two portions are not required to be planar. For example,the two portions may have curved surfaces. The two portions are notrequired to be in contact but may represent portions of virtual surfaces25 of two separate virtual objects 26 located in contact with or neareach other.

Case 4 of Tool Encountering a Hole

If one or more points of the virtual tool penetrate the virtual surface25 (step 149), the haptic rendering process 16 finds the surfacedirection vector 101 at the point of greatest potential penetration ofthe virtual object 26 by the virtual tool 28 (step 154). The hapticrendering process 16 then determines the penetration value at the pointof greatest potential penetration and saves this value for subsequentuse (step 156). The haptic rendering process 16 then attempts to movethe virtual tool 28 in the direction of the surface direction vector 101just determined (step 158). The haptic rendering process 16 then checksto see if no points of the virtual tool 28 at the proposed locationpenetrate the virtual surface 25 (step 160). If no points penetrate thevirtual surface 25 86, then the haptic rendering process 16 moves thetool 28 to the proposed location (step 121) and proceeds to step 120.This situation represents “Case 4,” which occurs if the virtual tool 28is in a corner, recess, or “hole” in the virtual object 26.

If one or more points of the virtual tool 28 penetrate the virtualsurface 25, then the haptic rendering process 16 proceeds to step 162and determines if the penetration values at all the tool points would beless than the previously determined penetration value at the deepestpotential penetration, as calculated in step 156. If all the tool pointswould be less than the previously determined penetration value, then thehaptic rendering process 16 returns to step 154.

If all the tool points would not be less than the previously determinedpenetration value, then the haptic rendering process 16 proposes movingthe virtual tool 28 in a random direction (step 166) from its previouspotential position and goes to step 149.

Description of a Voxel-Based Approach

In one embodiment of the invention, the virtual object 26 is implementedas a volume using concepts of voxels 78, density, and an isosurface 86.FIG. 9 shows several voxels 78 having different density values 0, 51,102, 153, 204, and 255, a gradient 80 established by these voxel values,and an isosurface 86. As used herein, density is not a physical propertyof the virtual object 26, but rather an abstract construct used fordetermining the shape of the virtual object 26. FIG. 10 illustrates apictorial view in two dimensions of the voxels 78 of a three dimensionalrectangular solid 82 with an isosurface 86 at value 128 indicating thesolid surface of the rectangular solid 82. As shown in FIG. 10, thevolume of the virtual object 26 is modeled using a 3-D array of evenlyspaced elements or voxels 78 located at discrete points in the virtualenvironment 26. In another embodiment, the elements are not required tobe arranged with regular spacing. Each voxel 78 stores a density value.Density values for points that lie between the discrete voxel points canalso be evaluated using interpolation. The volume also stores a densitythreshold value. Points having a density value greater than the densitythreshold are considered to be inside the object. Points having adensity value less than the density threshold are considered to beoutside the object. As used herein, an “isosurface” 86 refers to a locusof points all having an identical density value. In one embodiment, theisosurface 86 whose density value equals the density thresholdrepresents the virtual surface 25 of the virtual object 26. In oneembodiment, as shown in FIG. 10 this density threshold is 128 and thevoxel density values can range from 0 to 255. Thus a voxelrepresentation facilitates an easy method for determining whether pointson a virtual tool 28 lie inside or outside of a virtual surface 25 of avirtual object 26. All of the voxels 78 shown in FIG. 10 are associatedwith the rectangular solid 82, but a user moving a virtual tool 28toward the rectangular solid 82 would not encounter a solid feelingsurface until the virtual tool 28 contacts the isosurface 86. The userwould not feel any resistance to moving the tool 28 when moving throughthe voxels 78 with density values, such 0, 51, and 102, which are lessthan 128.

Implementation of the Haptic Rendering Process

The haptic rendering process 16 between a virtual object 26 and virtualtool 28 is described in more detail below for one embodiment of theinvention as implemented by the assignee, SensAble Technologies, Inc. ofCambridge, Mass.

The chosen volumetric representation is integrated with the GHOST® SDK(Software Developer's Kit) haptic interaction software developed bySensAble Technologies, which provides much of the necessary hapticsfunctionality and reduces haptic virtual objects into a set of basicmethods that are then handled correctly by GHOST® SDK. The GHOST® SDKuses the c++ programming language. The developer can create a high-levelobject which needs only to handle basic interactions such as determiningvectors, without being required to address low-level processes such asgenerating forces on the haptics device, resolving multiple collisions,and other more complex issues addressed by the GHOST software.

Volume Implementation Using Voxels

Haptic virtual objects are handled by a volume class. One embodiment ofthe invention is the gstVolume class. The gstVolume class follows thespecifications of the generally provided gstShape GHOST class andfollows the behavior of general geometric classes.

As described previously, the volume is represented using a voxel grid,the density values of which define an implicit virtual surface orisosurface 86 for the virtual object 26 as described for FIGS. 9 and 10.A valid volume for this representation is created containing anestablished gradient 80.

The specific voxel values defining the gradient 80 depend on the shapeof virtual object 26 being presented. In one embodiment, the voxelvalues may vary between 0 and 255, with the value 128 representing thevirtual surface. Any modifications to the volume must also preserve thegradient 80 to avoid incorrect calculation of surface direction vectors101 or penetration distances.

The voxel value at any point gives an indication of the penetrationdepth and the shortest distance between that voxel 78 and the surface 86of the volume. The ramp length is the number of voxels 78 over whichdensity values go from their minimum (0) to their maximum (255). Voxelvalues increase with the penetration depth until the values reach amaximum. All voxels 78 beyond that penetration depth are also set tothat maximum value. Likewise, when moving farther from the virtualsurface in a direction away from the volume, voxel values decrease untilthey reach the minimum value. All values in that direction beyond thatdistance are set to the minimum value.

The volume class is used to determine approximate surface directionvectors 101 to the surface 86 from any internal point or point along thesurface 86 of the volume using the density gradient 80 as explainedbelow. FIG. 11 illustrates a ramp length diagram comparing voxel valuesto penetration distances for one embodiment of the invention. FIG. 11depicts a vertical axis 198 for voxel values, a horizontal axis 202 forthe penetration distance, an isosurface line 205 representing a densitythreshold value of 128, a ramp length 206, a maximum voxel value 208with a value of 255, and a penetration line 210 at the isosurface 86corresponding to the density threshold value 128. The penetration line210 indicates that any penetration to the right of the line is into thesolid body of the virtual object 26.

Ramp Length

The maximum penetration depth where a gradient 80 exists is defined bythe ramp length 206, the density range, and the density threshold value.For example, for a ramp length 206 with a value of 4, a density range of0 to 255 and a density threshold of 128, any penetration beyond 1.99voxels [4/255*(255−128)] from the isosurface 86 will not lie within theportion of the volume where the voxel values exhibit a gradient. Assuch, the amount of penetration beyond this distance cannot be reliablycalculated. Beyond that, the voxel values will all be at the maximumvalue. In one embodiment, the maximum value is 255.

In one embodiment, the direction of the surface direction vector 101 iscalculated to be the direction of largest voxel density value gradientat that point. For any point that lies within the portion of the volumewhich exhibits a density gradient 80 a surface direction vector 101 canbe calculated which points to the virtual surface and whose magnitude isthe distance to the surface 86. In other words, in this region of space,the volume class can return a surface direction vector 101 which, whenadded to the current location, returns a location that is near thesurface 86 of the volume. The vector calculations contain the samelimitations as the penetration distance calculations; that is, thesurface direction vector 101 can only be calculated reliably within theportion of the volume which exhibits a density gradient 80.

Interpolating Values for Intermediate Points

Although the volume is characterized by a discrete array of voxels 78,it must be able to return a valid value at any point along thecontinuous range of its space, since the object it represents iscontinuous. For example, a line is characterized by two discrete points,but itself is continuous and can be evaluated at any point. If a valueis requested at a point where a voxel 78 is present, then thatparticular voxel density value is returned. For spaces between voxels78, the value may be resolved through tri-linear interpolation, a methodof converting a discrete set of points into a continuous field. Inanother embodiment, the interpolation can be based on other methods,such as a quadric interpolation.

In one embodiment, the evaluation is accomplished in a single step via amathematical formula that weights the contribution of each voxel 78 byits distance to the point being evaluated.

In one embodiment, voxels 78 are spaced one millimeter (that is, oneworld coordinate in a coordinate system based on millimeter spacing)apart. In other embodiments, the voxels 78 are spaced other distancesapart. In general, there is constant spacing between each of the voxels78, but there need not be for some embodiments. In further embodiments,memory management techniques may be utilized. A larger number of voxels78 can be represented, and thus either the resolution or size of theobject 26 can be increased.

Calculating Vectors

As stated previously, the required surface direction vector 101 at anypoint is determined using the direction of maximum density gradient atthat point. This maximum density gradient is determined using centraldifferences: the density value at a set distance from the point in thedirection of each Cartesian coordinate in turn is determined, and thedifferences between those values determines the vector direction.

FIG. 12 illustrates a method of calculating the surface direction vector101 of a point based on the density values of surrounding evaluationpoints. FIG. 12 shows a point 230 for which a direction vector 101 isbeing calculated. The surrounding evaluation points include a firstevaluation point 232 with a value of 40, a second evaluation point 234with a value of 100, a third evaluation point 236 with a value of 80,and a fourth evaluation point 238 with a value of 40. In one embodiment,each evaluation point is a voxel 78 and the values represent densityvalues.

The vector and density computations are used to project any point withinthe virtual object 26 to the virtual surface 25 of the virtual object26. Typically, this projection is used for the purposes of calculatingthe potential tool surface contact point 226. If a tool point 230 ispenetrating deeper than the region in which a gradient exists 206, thenno projection can be made directly from that point 230. Instead, thepoint must be first brought closer to the surface 25 (that is, to aregion where a gradient 80 exists).

FIG. 13 illustrates a region of a volume where a gradient 80 for avirtual object 26 exists, including a last tool point 224, a final toolsurface contact point 226, an initial proposed point 242, and a secondproposed point 244 for one embodiment of the invention. In FIG. 13, thehaptic rendering process 16 proposes to move the tool location such thata point 224 on the tool 28 would move to the point represented by 242.However, point 242 is not located within the region where a validgradient can be computed. Because the maximum surface direction vector101 cannot be calculated for this point, the haptic rendering process 16calculates a second proposed tool position that would result in the toolpoint 242 moving to the second proposed point 244. Point 244 is theapproximate midpoint of a line 246 between point 242 and the last toolpoint 224. If the voxel density value at that point 244 is still themaximum value (255 in one embodiment), the haptic rendering process 16can continue to average the point 244 with the last tool point 224, byusing a binary algorithm to bring it progressively closer to the lasttool point 224, until the midpoint 244 lies within the gradient 80. Oncewithin the region where a gradient exists, the surface direction vector101 to the virtual surface from the latest midpoint 244 can be computedfor the purposes previously described.

In one embodiment, a binary search method is used to determine theintersection point between the endpoints of the segment. The process isrepeated until either the points are within a tolerable error of thedesired value, or a maximum number of iterations has been reached.

FIGS. 14A, 14B, and 14C illustrate an example of a binary search forlocating the virtual surface 25 for a segment 249 that intersects thevirtual surface 25 for one embodiment of the invention. In FIG. 14A, thehaptic rendering process determines a midpoint 254 by averaging thelocations of the endpoints 248 and 252. In FIG. 14B the haptic renderingprocess 16 treats the point 254 as an endpoint of a modified segment 253extending from point 254 to endpoint 248. The haptic rendering process16 determines a midpoint 256 between points 248 and 254 and how farpoints 248 and 254 are from the virtual surface 25. The haptic renderingprocess 16 determines that point 248 is farther from the virtual surface25 than point 254, and thus sets the upper endpoint 248 to be at a newpoint 256. The endpoints of the additionally modified segment 257 arethus points 256 and 254. This process is repeated until a point is foundthat is regarded as the intersection point 258 (within a predetermineddistance from the virtual surface 25) of the original segment 249 andthe virtual surface 25, as shown in FIG. 14C.

Other Interaction Techniques

The following sections describe in more detail an implementation ofvirtual object 26 and virtual tool 28 interaction according to anembodiment of the invention implemented by the assignee, SensAbleTechnologies, Inc. of Cambridge, Mass.

Single Point Methods

In some contexts a single point virtual tool 28 may interact with thevirtual object 26. This may be handled as the simplest case ofmultipoint interaction using the methods previously described anddescribed in more detail later. In another embodiment, specializedmethods may be used for single point interaction, as described herein.

Collision Detection for Single Point Interaction

The haptic rendering algorithm determines when a collision between avirtual tool 28 and a virtual object 26 has occurred. A collision isdetected when the haptic rendering process 16 attempts to move a tool 28to penetrate a surface. In one embodiment, a collision occurs wheneverthe haptic interface location crosses through the virtual surface 25. Inone embodiment, the virtual surface 25 may be a NURBS surface. In oneembodiment, the virtual surface 25 may be the “skin” of a volumetricsolid.

A stateless haptic rendering algorithm would consider only the hapticinterface location 98 in determining the resulting forces from acollision. It would not consider any history or previous collisions. Theresulting forces in a stateless algorithm would use only the currenthaptic interface location 98 to determine the 1) depth of penetrationand 2) direction to the nearest surface 25. The resulting force vectorwould be a vector toward the nearest surface whose magnitude isproportional to the penetration depth. In the case of a voxelembodiment, the penetration depth of the haptic interface device pointis generally proportional to the voxel density at that point.

The direction of the force vector would be a vector that points from thehaptic interface location 98 to the closet point on the surface. In thecase of the voxel embodiment, this vector is simply the maximum voxelgradient (i.e. direction of greatest voxel density change) at the hapticinterface location 98. In conclusion, a stateless haptic renderingalgorithm would consider the location of the haptic interface device ateach iteration, determine if it has crossed a virtual surface 25 or isembedded in a virtual object 26, then return a force vector whosedirection is from the haptic interface device location 98 to the nearestpoint on the surface, and whose magnitude is proportional to thepenetration distance.

A stateless algorithm handles only basic cases of tool 28 interactionand falls short of accurately representing some cases. The most notablecase for which the stateless algorithm fails is for the case of thinobjects. If a user begins pressing through a thin surface, at some pointthe nearest surface point to the haptic interface device location 98will be on the other side of the thin object (i.e. after the user haspressed more than halfway through), and thus the force vector willincorrectly push the user out the other side.

An improved algorithm keeps track of a virtual tool position at eachiteration. Maintaining this virtual tool position is an efficient way tocontain state information about the history of the user's path ortrajectory. If the haptic interface device location 98 moves across asurface, the algorithm will attempt to move the virtual tool 28 towardthe haptic interface device location 98, but never through the surface,as if the two points were connected by a spring. The resulting forcesent to the haptic interface device 10 is proportional to the distancebetween the tool 28 and the haptic interface device location. In someembodiments, the force is also proportional to the difference invelocity or acceleration between the virtual tool 28 position and thehaptic interface device location 98. The tool position on the virtualsurface 25 is referred to herein as the surface contact point or SCP226.

In one embodiment, the haptic rendering process 16 attempts to minimizethe distance between the SCP 226 and the current haptic interfacelocation 98, given that a path of decreasing distance exists between thelast SCP 224 and the desired one. The connection between the SCP 226 andthe haptic interface location 98 can be viewed as a spring. The hapticrendering process 16 processes the locations of the SCP 226 and hapticinterface location 98 in iterative steps. At each iteration, the hapticrendering process 16 attempts to minimize the distance from the SCP 226to the haptic interface location 98 if possible.

In one embodiment, the haptic rendering process 16 uses an algorithm fordetermining the SCP 226 based on a stepping method. For a given numberof iterations, the algorithm determines a valid direction to move theSCP 226 which would yield a better solution (that is, decrease thedistance between the SCP 226 and the current haptic interface location98) and moves the point from the previous SCP 224 to a current SCP 226in that direction. In one embodiment, valid directions are those whichmove along the virtual surface 25 of the virtual object 26. The hapticrendering algorithm should not allow the SCP 226 to penetrate thevirtual surface 25 as it steps toward the haptic interface location 98.

FIG. 15 shows one method of determining a final SCP 226 used in oneembodiment. FIG. 15 illustrates changing SCP positions 264 a, 264 b, 264c, tangency planes 260 a, 260 b, 260 c, and resulting SCP 226. Theresulting SCP 226 is a stationary position until the user makes afurther movement of the virtual tool 28. In one embodiment, the changingSCP 264 is similar to the last or previous SCP 224. In one embodiment,the tangency planes 260 a, 260 b, 260 c are similar to the tangencyplane 104 described earlier. Using this process, the haptic renderingprocess 16 creates a tangency plane 260 a for an existing SCP 264 ausing the surface direction vector 101 a for the SCP 264 a. The hapticrendering process 16 then moves the SCP 264 a a fixed step(predetermined distance) on the virtual surface 25 to new SCP 264 b,constraining the movement of the changing SCP 264 b to the tangencyplane 260 b. The haptic rendering process 16 then determines a newtangency plane 262 b and repeats the stepping process to move the SCP264 b to a changed SCP 264 c. The haptic rendering process 16 thencreates a new tangency plane 262 c and repeats the process until the SCP264 c reaches a final position at SCP 226 above, or at the shortestdistance from, the haptic interface location 98, as shown in FIG. 15.This method of movement is also termed a march by the SCP 264 across thevirtual surface 25.

In one embodiment, the SCP 264 is constrained to move in the plane 260,which passes through the current SCP 264 and whose vector is theapproximate direction of the normal to the virtual surface 25 at thatpoint. In other words, the SCP 264 is allowed to move in any directionperpendicular to the surface direction vector 101 at that point. In oneembodiment, the haptic rendering process 16 determines which directionalong that plane 260 which would yield the optimum solution (that is,the solution that decreases the distance between the SCP 264 and thehaptic interface location 98 by the largest amount) and moves the SCP264 in that direction. Since the shape of the surface 25 at the SCP 264position may be curved, moving linearly in some direction may eithercause the SCP 264 to penetrate the virtual surface 25 (if convex) orleave the virtual surface 25 (if concave) causing an error in the SCP264 position. Therefore, the SCP 264 may be projected onto the surfaceat each step. In one embodiment, this error is minimal if the step sizeis small, so the haptic rendering process 16 only needs to project theSCP 264 onto the virtual surface 25 after the final position (finalsurface contact point 226) has been determined from the march. Anotherpotential problem involves local minimums and maximums. Because the SCP264 only moves in directions which draw it progressively closer to thehaptic interface location 98, the SCP 264 may not be able to traverselocal maximums (that is, small humps in the virtual surface 25) and maysettle in local minimums (small dips in the virtual surface 25). In oneembodiment, the solution is to allow the spring (a spring calculationconnecting the SCP 264 and haptic interface location 98) to stretch asmall and finite distance so that the SCP 264 march can overcome theselocal discrepancies. In one embodiment this problem of local maximumsand minimums rarely becomes a serious problem, so the spring stretchingapproach need not be implemented.

In another embodiment, the haptic rendering process 16 implements aniterative stepping algorithm as follows: The haptic rendering algorithmcreates a plane 260 passing through the current or changing SCP 264,whose surface direction vector 101 is the calculated approximate normalto the virtual surface at the SCP 264. The haptic rendering algorithmprojects the current haptic interface location 98 onto the nearest pointon that plane 260 a, 260 b, 260 c and creates a vector 262 a, 262 b, 262c from the SCP 264 a, 264 b, 264 c to that point. This vector thenbecomes the desired direction for the march. The haptic renderingalgorithm moves the SCP 264 a, 264 b, 264 c a fixed step in thedirection indicated by the vector 262 a, 262 b, 262 c. The hapticrendering algorithm repeats these steps. Finally, the haptic renderingalgorithm projects the SCP 264 a, 264 b, 264 c onto the virtual surface25 using the intersection technique described above. In an alternateembodiment, the haptic rendering algorithm uses the faster but lessrobust technique of projecting based on the surface direction vector 101and density value at that point. For example, see FIGS. 17A-17E.

Collision Detection with Three-Dimensional Tools

Collision with a point interface is sufficient for many interactions,but in one embodiment a more complete method of three-dimensional toolinteraction is used. The virtual tool 28 is represented by a series ofpoints along its surface, as discussed previously. At each iteration,the haptic rendering process 16 tests each of these points on thesurface of the tool 28 to test for penetration of the tool 28 into thevirtual surface 25 of the virtual object 26.

Improved Stepping Method

One embodiment of the haptic rendering process 16 maintains a toolposition and iteratively moves it toward the current haptic interfacelocation 98. The haptic rendering process 16 operates in a repeated loopof iterative processing steps. For each loop, the haptic renderingalgorithm attempts to minimize its distance to the haptic interfacelocation 98 without violating the virtual surface 25. In other words, ateach iteration, the spring connection between the tool 28 and the hapticinterface location 98 attempts to contract, but stops if that actionwould cause any of points of the virtual tool 28 to penetrate thevirtual surface 25.

During each iteration, the haptic rendering process 16 attempts to marchthe tool 28 toward the haptic interface location 98, checks forviolations against the virtual object 26, and repeats this process anumber of times. Since the haptic rendering process 16 evaluates forcollision at a potential tool location which is a distance equivalent tothe step size away from the previous tool location at each step, thestep size is bound to a relatively small value so that the hapticrendering process 16 does not jump the tool 28 over any features such asa concave “hole” in the surface 25 of the virtual object 26. The stepsize should also be large enough so that the movement of the tool 28 canreasonably keep up with the movement of the haptic interface location 98as the user moves the haptic interface device 10. If the hapticinterface device 10 moves a considerable distance but the step size ofthe tool 28 is small, the user feels a drag as the haptic renderingprocess 16 attempts to move the tool 28 to the haptic interface location98. In one embodiment, the step size for each iteration should be lessthan the minimum feature size that the user will want to feel. In onevoxel embodiment, the step size is one half of the voxel grid spacing.

Moving Along a Face of a Surface

If the movement method is to always move directly toward the hapticinterface location 98, the tool 28 would get “stuck” on the virtualsurface 25 once it touched the virtual surface 25. Therefore, the hapticrendering process 16 attempts to move the tool 28 along the surface 25in a manner that minimizes its spring distance (between the tool 28 andthe haptic interface location 98) instead of simply backing up to theprevious position whenever the tool 28 violates the surface 25. Themethods for achieving this are similar to those for determining the SCP226 for a single point.

FIGS. 17A-17E show a virtual tool 28 encountering a virtual surface 25and moving along the virtual surface 25 constrained by a plane 300. InFIGS. 17A and 17B, the tool 28 is not penetrating the virtual surface25, so the haptic rendering process moves the tool 28 to follow thelocation of the haptic interface location 98. In FIG. 17C the potentialtool position would result in penetration of the virtual surface 25 andthus violate the interaction constraints. The haptic rendering process16 determines the surface direction vector 101 at the point of greatestpotential penetration 304. In FIG. 17D the haptic rendering process 16determines a plane 300 perpendicular to the surface direction vector 101and passing through the last legitimate tool position in FIG. 17B. Thehaptic rendering process 16 constrains the virtual tool 28 to move onlywithin the plane 300. The desired direction of motion 306 is determinedby taking a vector 308 from a tool origin 27 to the haptic interfacelocation 98 and projecting that vector 308 onto the plane 300. In FIG.17E, the tool 28 moves a fixed distance in the direction determined inFIG. 17D. This manner of movement effectively allows the tool 28 to movealong the virtual surface 25 without getting “stuck.” Once the hapticrendering process 16 determines that the tool 28 is near the virtualsurface 25 (that is, after finding a potential penetration), this methodmay be used for all further calculations until the tool 28 leaves thevirtual surface 25 (that is, the potential tool points cease topenetrate the virtual surface 25, causing the tool 28 to no longer be ina touching state).

Thus, the haptic rendering process 16 can detect when the tool 28 is ona face of a virtual surface 25, and attempts to move it along thesurface 25 instead of directly toward the haptic interface location 98when this condition is detected.

Moving Along or Across an Edge

More complicated situations may occur where the haptic rendering process16 is moving the virtual tool 28 along an edge 108 or 177 instead of aface of a virtual object 26. In this situation, the haptic renderingprocess 16 constrains the tool 28 to a line (the edge) instead of aplane (the face). When multiple potential penetration points on thesurface of the tool 28 have differing surface direction vectors 101,then the haptic rendering process 16 assumes that the tool 28 is at anintersection of the two faces with differing surface direction vectors101, and constrains the tool 28 to that virtual edge 108 accordingly.The virtual edge 108 is the cross product of the surface directionvectors 101; the haptic rendering process 16 then constrains the tool 28to moving only in that direction or its negative. (See the discussion ofedge constraints associated with FIGS. 7A-7C.)

In one embodiment, it is not always the case that the haptic renderingprocess 16 constrains a virtual tool 28 to moving along an edge 108 or177 when one is detected. Otherwise, when attempting to slide acrossedges 177 that are convex (that is, mountains or “sharp” edges 177 inthe virtual surface 25 of the virtual object 26), the tool 28 may getstuck. Thus, the haptic rendering process 16 should distinguish betweenwhen the tool 28 is attempting to slide along a concave edge 108 versuswhen it is attempting to cross a convex edge 177. This is determined bytaking the dot product of the desired direction (that is, a vector fromthe tool origin 27 to the haptic interface location 98) with the surfacedirection vectors 101 of each face that forms the virtual edge 108 or177. See FIG. 7B. A positive dot product indicates that the tool 28 isattempting to move away from one of the virtual surfaces 25 instead ofsliding along the edge 108 or 177. If either dot product is positive,the haptic rendering process 16 assumes that the tool 28 is movingacross a convex edge 177 and does not constrain itself to the convexedge 177. When this case is detected, the haptic rendering process 16“pushes the tool 28 away” slightly, moving in the direction of thevector which is the average of the two surface direction vectors 101.

The methods described above are sufficient in most cases; however,scenarios exist where the movement of the tool 28 may get “stuck” andthe haptic rendering process 16 is unable to move the tool 28 along thevirtual surface 25 effectively. To compensate for this, the hapticrendering process 16 tries moving the tool 28 tries away from thesurface 25 (that is, in the direction of the surface direction vector101) at small increments if it has been stuck for a considerable time(that is, the haptic rendering process 16 has unsuccessfully attemptedto move the tool 28 for the past several iterations).

Progressive Extraction

Finally, there are situations in which the tool 28 somehow becomes stuckinside the virtual surface 25. In other words, the current tool positionis penetrating the surface 25. For example, if the user touches asurface 25 with the flat face of a cube, then rotates the cube such thatone of the edges is now penetrating the surface 25. Unless the hapticdevice limits this rotation via a torque feedback, the haptic renderingprocess 16 can put the virtual tool 28 in a position of penetration,which violates the desired behavior. In one embodiment, the hapticinterface device 10 would have six degrees of freedom of force feedbackand thus not allow invalid rotations. In other embodiments, the hapticinterface device 10 has more than six degrees of freedom. An embeddedtool position may also occur if the user uses a modification mode to addmaterial on top of the tool 28, or the virtual object 26 is rotated suchthat the tool 28 suddenly becomes embedded.

If the tool 28 is forced to an invalid position, the haptic renderingprocess 16 needs some way of extracting the tool 28 to a valid position.In these cases, the haptic rendering process 16 should attempt to movethe tool 28 in a direction away from the virtual surface 25 to escapethe surface 25. Therefore, if the tool 28 is stuck such that movingdirectly toward the haptic interface location 98, moving along theconstraint plane 300, moving to its previous position, and moving in thedirection in the normal of that plane 300 all would result inpenetration of the virtual object 26, then the haptic rendering process16 attempts to “jump” the tool 28 a significant distance in thedirection of the surface direction vector 101. (See FIG. 7C.) If thisjump still does not free the tool 28 (that is, one of its points wouldremain embedded in the surface 25), but the proposed position results inlesser penetration of the virtual object 26, then it is considered asuperior position (that is, if the point of greatest penetration yieldsa lower density evaluation value than the previous evaluation for theprevious point of greatest penetration). This condition allows thehaptic rendering process 16 to have a notion of “progress;” that is,even though moving in some direction does not fully release the tool 28from the surface 25, the movement is still an improvement over theprevious position if moving in this direction causes the penetrationdistance to decrease.

Random Extraction

The above methods handle most cases. However, some scenarios exist wherethe tool 28 is “stuck” but moving in the direction of the normal to theplane does not yield any improvement. This is a rare condition since thesurface direction vector 101 points toward the surface 25 of the virtualobject 26, moving in that direction should decrease the penetrationdistance. But although moving in that direction decreases thepenetration distance of that point which is used to determine thesurface direction vector 101, this movement may increase the penetrationof another point (usually one of the opposite side) such that the tool's28 overall penetration distance increases. In circumstances where noneof the above techniques yields acceptable results, the haptic renderingprocess 16 may conclude that the tool 28 is truly stuck and should notlegally move (for example, as in FIGS. 16A and 16B).

For example, suppose a spherical tool 28 ends up in a situation where itis in a gap or channel 270 between two virtual objects 272 and 274,whose width is less than that of the tool 28. A similar situation occursif the tool 28 is in a tunnel or conduit in a virtual object 26. FIGS.16A and 16B show a spherical virtual tool 28 in a channel 270 formedbetween two virtual objects 272 and 274. The diameter of the channel 270is less than the diameter of the virtual tool 28. FIG. 16A shows a pointof greatest penetration 276 and a vector 278 to the surface 280 ofvirtual object 274. FIG. 16B shows a new point with greatest penetration282 and its computed vector 284. The point 288 represents the toolsurface contact point for the previous penetration of virtual object274. The haptic rendering process 16 may determine that the point 276with the greatest X value has the greatest penetration (based on ahorizontal X axis for FIGS. 16A and 16B). It will then push the sphere28 toward the minus X direction by some distance. During the nextiteration, the point of greatest penetration will probably be the point282 with the least X value. This will then cause the sphere 28 to moveback in the X direction by some distance, etc. In this example, thehaptic rendering process 16 could cause the tool 28 to oscillateindefinitely between the positions depicted in FIGS. 16A and 16B.

Thus, as a final method, the haptic rendering process 16 may attempt tomove the tool 28 in a random direction as a way of arriving at a bettersolution (that is, one that decreases the greatest penetrationdistance). (See FIG. 7C.) However, in one embodiment, instead of movingin a purely random direction, the haptic rendering process 16 determineswhich of the points of the tool 28 have computed surface directionvectors 101 (that is are either embedded in the surface 25 or within thefield of decreasing non-zero values surrounding the surface 25) andchooses one of those surface direction vectors 101 randomly as thedirection to move in. The movement of the tool 28 obeys the same rulesas described for other cases in FIGS. 7A-7C. If the maximum penetrationdistance does not decrease as a result of the attempted move, then thehaptic rendering process 16 does not move the tool 28 but instead triesa different direction.

Note that this final method is fairly expensive computationally. Thus,when the tool 28 reaches this state, the haptic rendering process 16 maydetermine all the current tool penetration points only once, by creatinga list of indices of points whose penetration values are within thevirtual surface 25. In one embodiment using voxels 78, the pointsinclude those points with densities greater than the density of thethreshold value (isosurface 86). For all successive iterations, thehaptic rendering process 16 randomly chooses one of the points and movesthe tool 28 in the direction of that point's surface direction vector101.

Variable Step Size and Other Optimizations

Adaptive Step Size

In one embodiment, the user may feel a drag force if the hapticrendering process 16 is not able to maintain the movement of the tool 28with the movement of the haptic interface device 10. For example, if thestep size or number of iterations is small, then the haptic renderingprocess 16 may take a considerable amount of time to move the tool 28 toits final position if the haptic interface device 10 has moved asignificant distance. Thus, it is desirable to have an adaptivealgorithm that interactively sets the step size depending on themovement of the haptic interface device 10. If the haptic interfacedevice 10 is moving quickly, then the step size increases to compensate.If the haptic interface device 10 is not moving quickly, the step sizeis decreased so that the tool 28 can be placed with greater accuracy. Asmaller step size also helps prevent undesired buzzing or vibration ofthe haptic interface device 10. Buzzing can occur when the hapticrendering process 16 cycles the tool 28 between multiple positions,attempting to settle but never finding rest. See FIGS. 16A and 16B.However, if the step size is decreased, these oscillations become verysmall and barely noticeable or do not occur at all. On the other hand,if the user is moving a large distance, then oscillations and “buzzing”do not occur because the haptic rendering process 16 is attempting tomaintain the position of the tool 28 with the haptic interface location98 of the haptic interface device 10 instead of having the tool 28 stayin one position. Thus, if during a given loop, the haptic interfacedevice 10 is not moving rapidly, the step size of the tool 28 isdecreased.

Adaptive Number of Iterations

If the computer processor is not performing any other process-consumingtasks, then the number of iterations per second can be increased safely.The most intensive operations occur if the user is not only interactingwith the virtual object 26 but is modifying it in some manner. Forexample, if the haptic rendering process 16 is using the tool 28 toremove or add material, the calculations can consume much of theprocessing time. If the virtual object 26 is not being modified, thehaptic rendering process 16 assumes it can use a greater portion of theprocessing time and thus increases its number of iterations. In oneembodiment, the number of iterations is increased by a factor of two. Inother embodiments, other factors are used to determine the number ofiterations.

Tool Point Collision Detection Optimization

The processing requirement per step can also be decreased if not allpoints along the tool 28 are evaluated at each loop. Evaluating a largenumber of points to check for penetration of the virtual surface 25 bythe virtual tool 28 consumes a large amount of processing time. In oneembodiment, a faster method attempts to optimize this by choosing only aset number of points to evaluate during each iteration, depending onwhich points are closest to the surface 25. In a voxel embodiment,during the first step in each loop, the haptic rendering process 16evaluates the density values for the voxels 78 at all tool points andremembers the five points that evaluated to the greatest densities. Forsubsequent steps during that loop, it characterizes the tool 28 by thosefive points and thus does not perform any collision detection with anyof the other tool points. This gives a significant performance increase.In one embodiment, the average tool 28 may have around 80 tool points,so evaluating the only 5 of 80 provides a large advantage. When thisoptimization is used, the number of iterations per loop is doubled tofour, so that the haptic rendering process 16 only performs collisiondetection against every tool point every fourth step.

This can lead to situations where one of the tool points would violatethe virtual surface 26, if that point was not one of the original fivechosen during the first step. The haptic rendering process 16 however,does not usually place the tool 28 in situations where a valid positioncannot be found following an evaluation of a potentially illegal seriesof steps.

Adaptive Stepping Behavior

In one embodiment if the haptic rendering process 16 has been moving thetool 28 toward the haptic interface location 98 without encounteringintersections with virtual objects 26) for several iterations, then thehaptic rendering process 16 assumes that the tool 28 is in empty space.For future iterations, instead of stepping toward the haptic interfacelocation 98, the tool origin 27 is set to be coincident with the hapticinterface location 98 at each step. Thus, the user does not feel anyunwanted force effects. This continues until the tool 28 intersects witha virtual object 26, in which case the haptic rendering process 16 usesthe previously discussed stepping algorithms to increment the toolposition to a valid location. The number of legal steps needed beforethe tool 28 is set to the haptic interface location 98 may be defined tobe one hundred iterations. In other embodiments, the number of legalsteps may be less than or greater than one hundred iterations.

Optimize Transform Computations

For each collision detection evaluation between a tool point and thevirtual surface 25, the point is transformed from the local coordinatesof the tool 28 to the local coordinates of the virtual object 26. Thegreatest process consumption involves multiplying the point by theappropriate transformation matrix to account for the rotation of thetool 28 and virtual object 26. In one embodiment, instead of calculatingthis transformation for each loop, the points are only updated to thecurrent orientation every one hundred iterations. When the points areupdated, they are stored in a separate point array of local points,which is used to represent the tool 28 until it is again updated. Notethat only the rotation of the points is updated every one hundrediterations; the translation of the points is accurately updated everyloop to accurately reflect the potential position of the tool 28.

Additional Conditions Handling

Normally, the haptic rendering process 16 places the tool 28 to remainon the outside of any virtual object 26 with which it is interacting. Ifsome small portion of the tool 28 becomes buried in the virtual object26, the tool 28 can usually be extracted to the surface 86 using one ormore of the techniques described above. However, situations exist wherethe tool 28 may instantaneously become completely buried inside avirtual object 26. For example, if the user adds a large amount ofmaterial over the current tool position, or the user begins the programwith the tool position inside a virtual object 26, then the tool 28 isburied. If the tool 28 is ever inside a virtual object 26, the desiredbehavior is for the haptic rendering process 16 to gradually push thetool 28 away from the virtual object 26 in the direction of the nearestsurface 86. In another embodiment, the tool 28 reaches a state where itis completely or nearly buried in a virtual object 26, then no forcesare generated until the haptic rendering process 16 moves the tool 28 toexit completely the virtual object 26. It sets its position to thehaptic interface location 98 for subsequent iterations and generates nocollisions until the tool 28 reaches a position where every tool pointis outside of the virtual object 26. In a voxel embodiment, the tool 28is considered buried if its center evaluates to a density valueapproximate to the maximum density value (which is a value of 255 in oneembodiment).

Tool Behavior Under Object Transformations

The tool position is stored in the local coordinates of the virtualobject 26. Thus, if the virtual object 26 instantaneously changes scale,translation, or rotation, the tool position may jump, and the user feelsa sudden “kick back” in the haptic interface device 10. To prevent thisin one embodiment, a check is made during the first potential step ofthe tool 28 to determine if the transform of the object 26 has changedsignificantly since the previous loop. If this is true, then the toolposition is set to the haptic interface location 98, and therefore noforce generated. For example, if an object 28 were scaled upward, thehaptic rendering process 16 would bury the tool 28 in the virtual object26 instead of kicking the tool 28 to the surface 86.

Tool Position Error

The tool 28 is most often not directly on the virtual surface 25. Asdiscussed previously, the haptic rendering process 16 only moves thetool 28 in discrete steps and its movement is constrained from violatingthe surface 25. Thus, the tool 28 rarely settles directly on the surface25. This difference in position is not particularly noticeable when theuser is feeling the surface 25 with the virtual tool 28. Realistic feeldoes not depend on having the tool 28 always directly on the virtualsurface 25. In one voxel embodiment, step sizes are generally around 0.1grid space unit; thus, the user at most experiences an error of 0.1 gridspace units from the surface 86. This is below the threshold where itwould make a difference in the force felt by the user. In otherembodiments, the step size may have other values than the 0.1 grid spaceunit.

Modification Techniques

FIG. 18 illustrates a flowchart of the modification process occurringbetween a virtual tool 28 and virtual object 26. First, a virtual tool28 is determined or generated in virtual space that represents thehaptic interface device 10 that the user is manipulating in real space(step 400). In one embodiment, the haptic rendering process 16 generatesthe virtual tool 28. The virtual tool 28 includes a plurality ofdiscrete points. The user then selects a modification mode thatdetermines what kind of modification occurs to the virtual object 26 asa result of interaction between the virtual object 26 and the virtualtool 28 (step 402). The modification modes can include material removal,material addition, and other material modification modes. The materialremoval modes can include a remove, erase, and sandpaper modes. Materialaddition can include addition and push and pull modifications. Thematerial modification mode can include smoothing, mirroring, and othermaterial modification modes.

In step 404, sensors determine the location of a user in real space. Inone embodiment the user is manipulating a haptic interface device 10 andsensors determine the position of the haptic interface device 10 in realspace.

The modeling application 12 then determines the location of the discretepoints of the virtual tool 28 relative to the location of the virtualobject 26 (step 406). In one embodiment the haptic rendering process 16determines these locations. The haptic rendering process 16 thencalculates an interaction force between the virtual tool 28 and thevirtual object 26 based on the locations of the points of the virtualtool 28 and the location of the virtual object 26 (step 408). The userfeels the interaction force through the haptic interface device 10,which thus provides feed back to the user on the interaction of thevirtual tool 28 with the virtual object 26. In one embodiment, thehaptic rendering processor 16 provides the interaction force to thehaptic interface device 10. In one embodiment, the virtual object 26includes a virtual surface 25 and the position and orientation of thevirtual tool 28 is determined relative to the virtual surface 25 basedon the locations of the points of the virtual tool 28 compared to thevirtual surface 25. In one embodiment using voxels the virtual surfaceis a virtual isosurface 86.

The modeling application 12 then produces a modified virtual object 26by modifying the virtual object 26 based on the modification mode, theposition of the virtual tool 28, and the location of the virtual object26 (step 410). In one embodiment, the modification processor 20 producesthe modified virtual object 26. For example, if the virtual tool 28 isin a removal mode and the user is attempting to penetrate the virtualobject 26 with the virtual tool 28, then the modification processor 20removes material from the virtual object 26 that is in the path of thetool 28 as the user pushes the tool 28 into the object 26. If themodification mode is a sandpaper mode, and the user is rubbing thevirtual object 26 with the virtual tool 28 (as though rubbing wood withsandpaper), then the modification processor 20 removes a limited amountof material from the virtual object 26 that corresponds to the areabeing rubbed and the force and amount of rubbing that the user isapplying to the virtual object 26.

The modified virtual object 26 is then output from the system. In oneembodiment, the output is a modified visual image of the virtual object26 that is output to a graphics display 14 by the modeling application12 or graphics processor 22. In one embodiment, the output also includesa new or modified shape, which the user feels through the haptic device.The user then decides whether to continue with the modification process(step 414). If the user decides to continue in the same modificationmode, the user makes an additional movement of the virtual tool 28, andthe haptic rendering process 16 determines the new position of thevirtual tool 28 (step 406). The user may decide to select a differentmodification mode (step 416) and returns to step 402 to select the newmodification mode. If the user does not decide to make any furthermodifications, then the virtual object 26 may be displayed, output, orsaved to a disk, tape, or other data storage device (step 418). Outputmay include output or export to on an alternate file format or aprinting device or a device that provides a physical, real world modelof the virtual object 26.

Basic Modification Techniques

The modification process between a virtual object 26 and virtual tool 28is described in more detail below for one embodiment of the invention asimplemented by the assignee, SensAble Technologies, Inc. of Cambridge,Mass.

Voxel-Based Removal

As described previously, one embodiment uses a voxel model to define theisosurface 86. Modifications to the isosurface 86 are made indirectly,by modification of the voxel density values, based on the virtual tool28 location relative to the isosurface 86.

In one embodiment, the tool 28 itself stores voxel density values forvarious locations throughout its volume. In another embodiment, thedensity values are calculated only when needed and are not stored. For agiven location of the tool 28 in the haptic virtual environment, the“tool density values” are then compared to the density values of thevirtual object 26 at those locations. The density values of the virtualobject 26 are then modified based on the comparison with the tooldensity values and the modification mode. For example, if themodification mode was set to removal, when the virtual object densityvalues were found to be greater than those of the tool 28 at theidentical location, the virtual object density values would be set tomatch those of the tool 28. This has the effect of lowering the densityvalue of the virtual object 26 at that location, resulting in amodification to the isosurface 86 that appears as removal of material.

How the modification mode is set affects how voxel densities are definedinside the virtual tool 28. For the case where modification mode is setto removal, the voxel densities are set in the tool 28 such that theyare at the minimum value on the most interior portions of the tool 28and set at a maximum value beyond the virtual isosurface of the tool 28.The direction of density value increase is reversed when modificationmode is set to addition. That is, the voxel density values are greatestwithin the tool 28 and go to their minimum beyond the virtual isosurfaceof the tool 28.

Example Tools

Many different tool shapes are possible. In one embodiment, the voxeldensities assigned within the virtual tool 28 are made based on distancefrom a tool “center”. For example, in the case of a sphere tool and amodification mode of removal, voxel densities at the center of thesphere would be their lowest value. On any line emanating radially fromthis center, the voxel density values would increase, depending on theramp length, until they reached the isosurface value at the apparentsurface of the virtual tool 28. On this same radial line, beyond thevirtual tool surface, voxel densities would continue to increase untiltheir maximum value was reached. The tool, then, in this embodiment, isa “field” of voxel density values, which when moved around by the usertransfers its values, when appropriate, to the virtual object 26,resulting in modification of the object 26.

Note that it is possible to define virtually any arbitrarily shapedfield, using this approach, and thus have an unlimited number of virtualtools 28. In one embodiment, the user can draw arbitrary 2-dimensionalshapes which are then populated with voxel density values, much like themore complicated 3-dimensional example above. In one embodiment, theactual shape of the tool is created by carving the virtual object 26,and then defining the shape of the tool 28 to be that of the object 26.These interactively carved tool shapes can be saved to a file and usedlater or shared through the Internet.

Pillboxing

Although the user is conceptually moving in a continuous motion, toollocations are only sampled discretely. As the user can move the virtualtool 28 quite quickly through space, it is necessary to be able toreliably remove or add material on the path between sampled toolpositions. FIG. 19A provides a pictorial view of a removal based on aremoval line 454 between two such points P21, P22. FIG. 19A is a 2-Drepresentation of a 3-D removal process. Conceptually, planes 456 and458 are formed at each endpoint and extend vertically out of the planeof the FIG. 19A. The removed area may be termed a “pillbox” or sweptarea 452 between the two points P21, P22. In one embodiment, this typeof removal may occur automatically when the user holds down a switchbutton on the haptic interface device 10 such as a stylus having abutton on it, and the user moves the stylus to drag the haptic interfacelocation 98 through a region to form the swept area 452. In anotherembodiment, the haptic rendering process 16 may receive a request thatis equivalent to sweeping a sphere 34 across an area, in effect creatinga cylinder with spherical ends.

FIG. 19B provides a pictorial view of endpoints P21, P22, a point P23where the appropriate voxel density value must be computed, a projectedpoint P24, and conceptual endpoint planes 456, 458. In FIG. 19B avirtual tool 28 is represented by sphere 462 a representing the startingposition of the tool 28 and sphere 462 b representing the endingposition of the same tool 28. Two planes, 456, 458 exist conceptually,one at each endpoint of the segment 454. If the point of interest, P23,lies between the planes 456, 458, then the point P23 is projected theshortest distance to the line segment 454. The resulting density valueat point P23 is based on the distance from point P23 to the line, as ifthe tool 28 was centered at the point P24.

Optimizing Removal and Addition

When the user requests a removal by moving the virtual tool 28 in aremoval mode, the haptic rendering process 16 determines the number ofvoxels the tool 26 currently occupies, the desired settings for each ofthese voxels, and then changes these voxel values if needed. If thehaptic rendering process 16 determines that the current voxel value atsome point is less than the desired voxel value, then the hapticrendering process 16 does not need to modify that point. For removal,only voxels that need to be decreased (that is, voxels whose densitiesare greater than the desired target density) are modified.

In one embodiment, the feel of the object 26 must be updated at a veryhigh rate. As such, removals are made at a slower rate than the feel ofthe object 26 is updated. Whenever a removal request is read, a boxenclosing the extent of the removal operation is determined, and thenumber of voxels in that box is calculated. The removal is then spreadover several iterations. Because the removal extends over multiplecycles but touch is performed each cycle, the virtual object 26 may notfeel “correct” to the user during an intermediate stage of the removal.However, in practice, this difference is small and offset by the benefitof allowing large-scale removal.

Ensuring Accurate Modification and Feel

Although the tool 28 is frequently intended to be “just touching” thevirtual surface 25, in practice it is usually only approximatelytouching. As such, the first step in modification is to accuratelyposition the tool 28 as just touching the virtual surface 25. In oneembodiment, this step is accomplished by doing a binary search along theapproximate surface normal at the point of greatest potential toolpenetration. Using the binary search, a more exact location for thesurface 25 at the potential penetration point is computed, and the toolorigin 27 is then moved such that the point in question is coincidentwith the computed surface location.

Since the point locations used for sensing penetration may have spacebetween them, it is possible for the sample points to not penetrate thevirtual surface 25, but still have potential embedding of the virtualtool 28. FIGS. 20 and 21 illustrate two cases of this. FIG. 20 depicts acube 466 near a virtual surface 25 which has a protruding feature 490 ofthe virtual surface 25 penetrating between two points, P31, P32 of thecube 466. In FIG. 20, the features are more detailed than the pointdistribution on the cube 466, so the feature 490 is able to penetrateundetected between the points P31, P32. FIG. 21 depicts a sphere 34penetrating a virtual surface 25 between two points S3, S4 of the sphere34. The sphere 34 is a convex object, so penetration may occur even whenthe virtual surface 25 is completely flat.

In the case of a tool 28 that is a sphere 34, the haptic renderingprocess 16 can reliably determine how much penetration has occurredbetween points of the sphere 34 when contacting a reasonably flatvirtual surface 25. For example, FIG. 22 depicts a sphere 34 representedby multiple points S11, S12, S13, S14 none of which penetrate a virtualsurface 25. FIG. 22 also shows a surface direction vector 101 at pointS12, a vector 494 from S12 to the origin 496 of the sphere 34, andcalculated distance 498 from the origin 496 to the virtual surface 25.At the current contacting point S12, the haptic rendering process 16first finds the surface direction vector (v1) 101 of the virtual object26 at that point. The haptic rendering process 16 then takes the dotproduct between the normalized surface direction vector (v1) 101 and thevector (v2) 494 from the point S12 to the origin 496 of the sphere 34.The resulting distance 498 indicates how much of the sphere 34 is notpenetrating the virtual surface 25. This distance is then used incorrecting the position of the sphere 34 to be tangent to the virtualsurface 25. Note that this position correction is only performed tocalculate the tool position for modification. When the user is simplyfeeling the virtual surface 25, no correction to the tool position needsto be made because the feel of the tool 28 does not have to beexceptionally accurate to create a compelling feel to the user.

The second step in modification is to then tentatively embed the tool 28by a specified penetration distance based on the removal metaphor ormode. The embedded tool 28 is then superimposed against the virtualobject 26 and, in one embodiment, the voxel densities of the virtualobject 26 updated based on the voxel densities of the tool 28 at thesuperposition locations. The penetration rate can be at a constant rateor it can be at a variable rate based on equations, functions, or otherrules. For example, a rule or equation could state that no modificationoccurs until a certain force threshold is met or that the amount ofpenetration is dependent in a non-linear way on the amount of forceapplied or the velocity of the movement.

As the virtual object 26 is updated, both the graphical display and thehaptic feedback or touch sensation are updated at very high rates. Byproperly limiting tool motion, using concepts such as softness,friction, and many other physical simulations, virtually any effect canbe simulated for the user, including the effect of feeling the surface25 while carving.

Modification with Alternate Feel

Although it is frequently desirable to give the user the feeling that heis carving a surface 25, it is also possible to give the user othertouch sensations when performing modifications. For example, anon-directional viscous force can give the sensation of pushing the tool28 through a thick liquid. In one embodiment, when the user activatesthe stylus switch on the haptic interface device 10 to begin removal, apoint constraint is generated at the haptic interface location 98. Thispoint follows and matches with the haptic interface location 98 unlessit is removing material. (In other words, if the haptic interfacelocation 98 is surrounded by empty space, the user does not feel anyforce effects). When modification occurs, this point constraint movesmore slowly toward the haptic interface location 98 and thus produces adrag effect. The force the user feels is modeled as a spring effectbetween the point constraint and the actual haptic interface location98.

To determine if any material needs to be removed, the haptic renderingprocess 16 performs the removal method and notes the number of voxelsthat have been affected. If this number is greater than zero, then thehaptic rendering process 16 determines that the tool 28 is removingmaterial and applies a drag force to the tool 28. The distance the tool28 is allowed to move during removal is dependent upon the user-definedremoval rate.

In one embodiment, using the voxel approach for more accuracy, thisremoval rate can be dependent upon the number of voxels changed, so thatthe tool movement is further constrained if a larger number of voxelshave been affected. Conversely, if the haptic rendering process 16determines that the movement of the tool 28 only removes aninsignificant number of voxels, then it can be permitted to movefurther. In one embodiment, the approach is to count those voxels thathave been changed from above to below the isosurface value 86. Inanother embodiment, a force threshold is imposed, such that the hapticrendering process 16 would only remove material if the threshold exceedsa certain user-defined force.

Addition

As mentioned previously, addition is the opposite of subtraction: thesame methods are used except for some minor changes to reverse thesubtraction process. In one embodiment based on the voxel approach, thegradient within the tool 28 is inverted. That is, the higher voxelvalues are found in the most interior region of the tool 28 and thelowest values outside of the virtual surface of the tool 28.

Existing voxels are only changed to the desired voxel values when thelatter exceeds the current values. Force effects are also different. Asvoxels are being added, the user feels only a 3-D-friction force similarto that experienced in non-contact removal mode. In other words, a pointconstraint is activated, and this constraint is permitted to move towardthe haptic interface location 98 at a set but slow rate. Thus, the userexperiences a drag while adding material.

Sandpaper

In one embodiment, one removal mode is the sandpaper removal metaphor.When sanding, the objective is to remove small amounts of material at aconsistent depth. One way to accomplish this is using outriggers, to bediscussed later, which would constrain the tool 28 to the virtualsurface 25 of the virtual object 26 outside the area currently beingsanded. Other embodiments utilize other methods. Regardless of mode, thesander (a virtual tool 28 in sandpaper removal move) only removesmaterial when the haptic interface location 98 velocity tangent to thevirtual surface 25 has exceeded a certain threshold.

A frictional effect is introduced while the virtual tool 28 iscontacting the virtual surface 25, regardless of whether removal isbeing currently performed or not, thus representing the real-world feelof a sander. When the sander is not removing, the frictional effect isconstant. When the sander is removing, the frictional effect oscillates.This oscillation makes the virtual surface 25 feel somewhat “rough” tothe user, as is expected for a sanding tool.

In one embodiment, the actual area that is sanded is determined bytaking the tool 28, shrinking it, and then penetrating it into thevirtual surface 25 of the virtual object 26 at a set distance. Forexample, in the case of a cube 466 the actual sanding element is asmaller cube protruding from the cube 466 that the user touches thevirtual surface 25 with. In another embodiment, the sander isimplemented as a 2-D blade that protrudes from the surface of the tool28.

In one embodiment, the current sandpaper metaphor attempts to smooth thevirtual surface 25 while avoiding gouging. The sandpaper method keepsthe tool 28 on the virtual surface 25 and only allows the tool 28 tomove laterally along the plane of the virtual surface 25. As discussedpreviously for normal modification, the tool 28 is first placed directlyon the virtual surface 25, then embedded a certain distance toward thehaptic interface location 98 depending on the removal rate. Thesandpaper method places the tool 28 directly on the virtual surface 25,then restricts the tool 28 to moving along a plane defined by the normalat the virtual surface 25.

Thus, the desired direction (that is, a vector from the tool 28 to thehaptic interface location 98) is projected onto the plane, and the tool28 is moved in that direction. Then the removal is performed based onthe movement of the tool 28.

Pull Mode

Pull mode is an interaction mode that reverses the virtual surface 25and binds the user inside the virtual object 26.

In one embodiment, using a voxel approach where the maximum density isrepresented by 255 and the minimum density is represented by 0, alldensity evaluations become invented to 255 minus the pre-inverted value.Thus, empty space (0) becomes maximum density space (255), and viceversa. The effect is that the haptic rendering process 16 calculates thetool 28 interaction with the inside of the virtual object 26 instead ofthe outside.

All removal and addition methods work effectively in pull mode. However,adding material in this mode feels to the user as if she is pulling thevirtual surface 25 outward. Likewise, the feel for removing material isthe opposite. Thus, when in pull mode, all methods are reversed. Theforce effects for removal become those that the user would otherwisefeel when normally adding. Likewise, the force effects that the userfeels while normally adding would be used for pull removal. In oneembodiment, the user feels a 3-D friction force while adding. Thus, inpull mode, the user feels this friction force while removing.

Non-Cutting Tool Geometries

Portions of a tool may be declared to be non-cutting. This means thatthey may be used to feel the object or limit the tool motion, but theydo not actively modify the virtual object. For example, consider asphere tool with a plane through its center, with the sphere declared ascutting and the plane declared as non-cutting. In this example, the usercould only embed the sphere in the material until the plane touched thematerial, limiting the cut. Another example of this is “outriggers,”where the non-cutting portion flanks the cutting portion of the tool.

FIG. 23A shows an outrigger construct 530 including arms, showngenerally as 532, and outriggers, shown generally as 534, connected tothe handle 32 of the sphere 34 of a virtual tool 28. When used in amodification mode, such as sandpaper, the user may use the virtual tool28 to remove material from an object 26 at a prescribed depth only. Thedepth of material removed is determined by the position of theoutriggers 534 a, 534 b where they touch the virtual surface 25 of thevirtual object 26. That is, the sphere 34 is allowed to penetrate thevirtual surface 25 and remove material from the virtual object 26, butthe outriggers 534 a, 534 b are not allowed to penetrate the virtualsurface 25. The outriggers 534 a, 534 b are attached to the handle 32 ofthe virtual tool 28 by rigid arms 532 a, 532 b. The rigid arms 532 a,532 b thus limit how far the sphere 34 of the virtual tool 28 penetratesthe virtual object 26 and how much material the modification process 20removes. The user can adjust the length of the rigid arms 532 a, 532 bto adjust the depth of penetration of the sphere 34 into the virtualobject 26. In one embodiment, the outrigger approach is implementedusing one rigid arm 532 and one outrigger 534 (not shown in FIG. 23A).In further embodiments, more than two arms and outriggers are used (notshown in FIG. 23A). The outrigger approach is not limited to a virtualtool 28 having a handle 32 and sphere 34, but may also be applied to avirtual tool 28 having other geometric shapes.

FIG. 23B depicts an outrigger construct 530 showing the rigid arms 532a, 532 b connected to the handle 32 of the tool 28 by a slip ring 538,for one embodiment of the invention. The slip ring 538 provides for someslippage, by a predetermined factor, of the outrigger construct 530 asthe sphere 34 penetrates the virtual surface 25 of the virtual object26. The user feels resistance as the slip ring 538 slides along thehandle 32, as the sphere 34 is allowed to penetrate further into thevirtual object 26. Typically, the resistance provided by the slip ring538 is a constant resistance force as the sphere 34 moves into thevirtual object 26.

FIG. 23C depicts an outrigger construct 530 connected to the handle 32by a spring 540, for one embodiment of the invention. The user feels aspring-based feedback force from the spring 540 that indicates how farthe sphere 34 is penetrating into the virtual object 26. In other words,as the tool 28 penetrates farther into the virtual object 26, the userfeels an increasing resistance to movement into the virtual object 26due to the feedback from the spring 540.

FIG. 23D depicts an outrigger construct 530, showing a combinationconnector 542 of a spring 540 and slip ring 538, for one embodiment ofthe invention. In this embodiment, the combination connector 542provides a springlike feedback force of increasing resistance topenetration of the virtual object 26 until a predetermined limit isreached, and then the combination connector 542 slips along the handle32 in a slip ring 538 effect. In one embodiment, after the predeterminedlimit is reached, the user feels a steady resistance to furtherpenetration into the virtual object 26, as the slip ring 538 effectcontinues. In another embodiment, this slippage is for a predetermineddistance, and then the combination connector 542 restores a springlikeeffect.

Smoothing

Smoothing blurs large features and eliminates some smaller features on avirtual object 26. The resulting appearance of the virtual surface 25 ofthe virtual object 26 is similar to the appearance of a wax model thathas been partially heated. Smoothing is important for removing bumps andsmall undesired detail.

Smoothing can be applied to portions of the virtual object 26, or thevirtual object 26 in its entirety. The user selects a region of thevirtual object 26, which is indicated by the region being highlighted(that is, drawn) in a different color.

In one embodiment, the user paints the virtual surface 25 of the virtualobject 26 with the virtual tool 28 as a way of selecting the region ofthe object 26 to be smoothed.

In one embodiment using the voxel approach, selection smoothing works byallowing the user to paint an area on the virtual object 26 to besmoothed. FIG. 24 illustrates a two dimensional view of a grid 489 ofvoxels 78, showing a painted area 493 of the surface 86, correspondingvoxels 495 that correspond to the painted area 493, and a selectionvolume 497 of voxels 78 including ramp. The selected area 493 is thensmoothed at the voxel level and an isosurface 86 is recomputed from thenew voxel values. The set of voxels spanning the isosurface 86 and voxelramp corresponding to the area selected become the selection volume 497.The selection volume 497 is then filtered through an N×N×N kernel inthree separate, one dimensional passes. Each successive pass uses theresults from the previous pass to accumulate the final resulting voxelvalues. Each source voxel in the selection volume 497 is filtered usinga kernel of size N, where each element of the kernel can vary accordingto the smoothing result desired. A kernel of size N results in thesource voxel being filtered against N/2 (rounded to the nearest integer)voxels 78 in each dimension, X, Y, & Z.

The resulting filtered selection volume 497 is then copied back into thesource volume and a new isosurface 86 is computed using the new voxelvalues. The number of voxels 78 remains constant in the volume, but thedistribution of voxel values may change, resulting in either more orfewer surface triangles in the resulting isosurface 86.

Higher smoothing levels correspond to an overall smoother shape and aloss of detail in finer features, whereas lower smoothing levelscorrespond to less loss of detail and a more localized smoothing effect.

Geometric Constraints

The haptic rendering process 16 may constrain or limit the motion of atool 28 along one or more degrees of freedom using a force feedbackapproach. Constraints may take the form of geometric entities,equations, rules, or other methods of defining a restriction to toolmotion. For example, a rule might state that the tool may only remove acertain amount of material per cut invocation. Another example oflimiting tool motion is a rule or equation which states that the motionof the virtual tool is to emulate the motion of the haptic device inphysical space, but as a reduced “gain” for example, reducing the amountof motion by a factor ten. Tool constraints may limit the ability of thetool 28 to move, translate, and or rotate in virtual space. In addition,geometric constraints may be associated with point, line, curve,surface, or space representation. For example, FIG. 25 illustrates avirtual tool 28 moving through a virtual object 26, but constrained by ageometric constraint, which in this case is a curved line 550 to move ina direction indicated by vector 552. For example, the user may be usingthe tool 28 to remove material from the virtual object 26, but wishes toremove the material in only a certain manner. Therefore, she sets ageometric constraint, such as the curve 550 shown in FIG. 25, thatlimits how the tool 28 moves, thus affecting what material is removed.

Geometric constraints can either restrict the user's movements by“holding” her on the geometry or by blocking her from the geometry. Foran example of a blocking constraint, consider a person who moves herfingertips along a tabletop, with the result that the fingertips areconstrained only to move along the plane of the table. The fingertipsmay be lifted off of the table, but they cannot be go through the table.For a virtual tool 28 limited by a blocking constraint by a plane, thehaptic rendering process 16 allows the tool 28 to touch the plane andmove away from it, but does not allow the tool 28 to penetrate theplane.

An example of a hold constraint is one that keeps the fingertips on thetable top, but would not allow the fingertips to move off of thetabletops. For a virtual tool 28 limited by a hold constraint to aplane, the haptic rendering process 16 allows the tool 28 to touch theplane, but does not allow the tool 28 to penetrate it or move away fromit depending on whether the constraint is permeable or not, as discussedbelow. In one embodiment, the user receives a force feedback impressionthat the tool 28 is “stuck” to the geometric constraint, in thisexample, the plane. Referring to FIG. 25, if there is a hold constraintto the curve 550, the user is given the feeling that the tool 28 isstuck to the curve 550.

Geometric constraints may also optionally have a force-field surroundingthem. For example, a hold constraint might “pull” a user onto it whenthey are near it. Or a block constraint might push the user away as theyapproach it.

Geometric constraints are permeable or nonpermeable. For a permeableconstraint, the user may make the tool 28 break through or break awayfrom the constraint by applying a force beyond a threshold. That is, ifpermeable, the user can set the geometric constraint to be penetrated bythe virtual tool 28, for example, with a sufficient exertion of force bythe user. A nonpermeable constraint does not allow any departure fromthe limitations that they impose on the position or orientation of thetool 28.

Constraints can also be placed interactively by using the hapticinteractive device 10, with the user feeling a force feedback effectwhen doing so. Thus the user can place a geometric constraint at orwithin a virtual object 26 using the haptic interactive device 10. Theuser then uses the haptic interactive device 10 to control a virtualtool 28 to modify the virtual object 26, with the movement of thevirtual tool 28 limited by the geometric constraint. In anotherembodiment, the user makes a constraint, such as by using the virtualtool 28 to inscribe a slight groove in the virtual surface 25 of avirtual object 26. The user then uses this groove as a geometricconstraint, when using the virtual tool 28 to enlarge the groove to makea larger indentation in the virtual surface 25 based on the shape of thegroove.

In one embodiment, the virtual constraint can take the shape of anygeometric shape, including a previously generated virtual object orportion of a virtual object. Virtual constraints derived from virtualobjects are referred to herein as bucks. Bucks may be visibly distinctfrom the virtual object being carved. Bucks can be particularly usefulfor sculpting product designs that must contain certain objects within.For example, if a toy is to contain batteries, the designed canintroduce bucks that represent the batteries into the virtualenvironment, thereby limiting himself to sculpt only portions of thevirtual object that lie outside of the volume of the batteries. Becausethe bucks make the region of the model occupied by the batteriesunsculptable, the design sculpted by the designer is guaranteed tocontain a volume large enough to enclose the batteries.

Portions of the virtual object being sculpted can also be designated asvirtual constraints, using the painting selection method describedpreviously for smoothing. Regions of the model that are painted by thedesigned are thereby designated unsculptable. This feature preventsaccidental modification of finished portions of a virtual object, whileallowing modification of unfinished portions of the virtual object.

In an embodiment that uses voxels, each voxel has a flag whichrepresents sculptable or unsculptable. Bucks are achieved by settingthis flag to unsculptable for each voxel that lies within the region orshape of the buck. This modification process will not substantiallyalter the value of the voxels designated as unsculptable.

In one embodiment, the tool 28 is constrained based on the constraintgeometry, the position of the virtual object 26, the position of thevirtual tool 28, and the haptic interface location 98. In anotherembodiment the tool 28 is also constrained by the virtual surface 25.For example, if a user is removing material from a virtual object 26,the user feels a force feedback or drag when moving into the object 26and removing material. In addition, if the virtual object 26 isconstrained by an impermeable hold constraint, as in FIG. 25 to a curve550, then the user also feels a force feedback that makes the user feelthat the virtual tool 28 is stuck to moving only along the curve 550.The user is allowed to remove material in the virtual object 26 whenmoving along the curve 550, but is not allowed to move away from thecurve 550 and to remove material elsewhere in the virtual object 26 inan area where the virtual tool 28 is not touching the curve 550.

FIG. 26 is a flowchart illustrating the process of using a geometricconstraint to constrain the virtual tool, for one embodiment of theinvention. First, the user generates an initial or starting virtualobject 26 having a virtual surface 25 in a haptic virtual environment(step 600). The user then selects a geometric constraint that determineshow a virtual tool 28 will be constrained when it moves in the hapticvirtual environment (step 602). A virtual tool 28 is determined orgenerated in virtual space that represents the haptic interface device10 that the user is manipulating in real space (step 604). In oneembodiment, the haptic rendering process 16 generates the virtual tool28. In step 606, sensors determine the location of a user or designer inreal space. In one embodiment the user is manipulating a hapticinterface device 10 and sensors determine the location of the hapticinterface device 10 in real space.

The modeling application 12 then determines a haptic interface position98 in the haptic virtual environment that represents the location of thehaptic interface device 10 in real space (step 608). The modelingapplication 12 then determines a proposed position of the virtual tool28 in the haptic interaction space in comparison to the haptic interfacelocation 98 and the location of the virtual surface 25 (step 610). Nextthe modeling application 12 determines if the proposed position of thevirtual tool 28 penetrates the virtual surface 25 (step 612). Themodeling application 12 constrains the action of the tool 28 based onthe constraint geometry, the virtual surface 25, the position of thevirtual tool 28, and the haptic interface location 98 (step 614). In oneembodiment, the modeling application 12 constrains the translation orrotation of the tool 28. In another embodiment, the haptic renderingprocessor 16 constrains the action of the tool 28. If the tool 28 doesnot penetrate the virtual surface 25, the process checks the location ofthe user in real space (step 606) to determine if the user has moved thehaptic interface device 10, thus causing a change in the hapticinterface location 98 (step 608). If so, steps 610 and 612 are repeated.

Virtual Object Orientation

The user can interactively change the position and orientation ofvirtual objects 26 during evaluation and modification operations. In oneembodiment, the user uses a 1 DOF (degree of freedom) or greater inputdevice such as a mouse or trackball to directly affect the position andorientation of the virtual object about an arbitrary point. In oneembodiment the user uses the input device using the non-dominant hand.In another embodiment, the user uses a 1-DOF or greater haptic device,such as a haptic interface device 10, to directly affect the positionand orientation of the virtual object about an arbitrary point. In afurther embodiment, the user uses keyboard commands to directly affectthe position and orientation of the virtual object 26 about an arbitrarypoint. While manipulating the virtual object with the non-dominant hand,the user has the option of making modifications with the virtual tool.That is, while the virtual object is in motion, whether through activemanipulation or some other means such as an inertia effect, modificationmay be effected with the virtual tool either in motion, or in astationary position.

Virtual Control Wheels

In one embodiment, the user also uses one or more virtual control wheels586 to control the orientation of the virtual object 26. FIG. 27 shows avirtual object 26 controlled by a control wheel 586 and a virtuallathing tool 588. The user uses any virtual tool 28 to touch and movethe control wheels. In one embodiment, the user uses a virtual lathingtool 588 to modify the virtual object 26. The user uses the virtuallathing tool 588 to remove material, in a manner similar to a cuttingwood on a physical lathe, as shown in FIG. 27. The user can also addmaterial to a virtual object 26. In addition, the user can start therotation of the control wheel 586 and add material from a virtual tool26 when acting in an extrusion, addition, or “toothpaste” mode. In thiscase, the added material may form a circular or spiral shape as extrudedfrom the virtual tool 26 into the rotating space controlled by thecontrol wheel 586.

The control wheels 586 control the orientation of the virtual object 26about an arbitrary axis. FIG. 27 shows the control wheel 586 controllingthe rotation of a virtual object 26 about a vertical or z axis. In oneembodiment, the user can use control wheels 586 to control the rotationof a virtual object 26 about the x axis and the y axis also. In oneembodiment, the user can use control wheels to control the rotation ofthe virtual object 26 about two or more axes at the same time. Forexample, the user uses two control wheels to cause the virtual object 26to rotate about the x axis and z axis simultaneously and use a virtuallathing tool 588 or other virtual tool 28 to modify the virtual object26 while it is rotating about two or more axes.

The control wheels 586 may have properties associated with mass,damping, surface compliance, and friction.

In one embodiment, the control wheels 586 are programmed to stop as soonas the user stops interacting with them or to continue rotating, (thatis, much like a physical lathe) until they are explicitly stopped.

In one embodiment, the user rotates the control wheels 586 withoutconstraint. In another embodiment, the user constrains the rotation toan angular extent.

In one embodiment, the control wheels 586 have haptically sensible“indentations” associated with their rotations, much like some realworld, physical rotary knobs.

3-D Sketch Modeling

The 3-D sketch system is a system that allows rapid and intuitivecreation of 3-D shapes. These shapes are generated from planar sketches.The user of the 3-D sketch software sketches outlines or profiles of theshape they desire from different views of the object 26 (for exampletop, right and front views). These sketches are then used to “cut” thedesired shape out of a virtual block of material or virtual object 26.In one embodiment, the user creates the outline sketches on constructionplanes, referred to generally as 700 (see FIG. 28A). The user can feeland manipulate the planes 700 using a 3-D force feedback device orhaptic interface device 10. Through the haptic interface device 10, theuser controls a virtual drawing tool 690 which, in one embodiment, canbe a pen used to feel and create sketches on the planes 700. In oneembodiment, the virtual drawing tool 690 includes a longitudinal portionor handle 692 and an end 694. In another embodiment, the end 694includes a drawing point used to create the sketch 702. In oneembodiment the haptic interface device 10 is a stylus with a switch orbutton on a forward part of the stylus where it can be readily pressedby one of the user's fingers. In another embodiment, the end 694 of thevirtual drawing tool 690 is a virtual hand used to feel, select andmanipulate one or more planes 700 and the block of material or virtualobject 26. However, in other embodiments the user can use the 3-D sketchapproach without any force feedback effect. That is, the 3-D sketchsystem does not require a force feedback approach.

The process is similar to creating models with blue foam and a hot wire,although there are a number of ways in which the 3-D sketch system cando more than the physical process. Generally, when designers create bluefoam models they create sketches that they attach to the faces of ablock of blue foam. They use these sketches to guide the hot wire as itcuts through the block of blue foam. For any given cut, they use onlyone sketch. In one embodiment of the invention, in addition to placingsketches on the outside of the block, the user can also place sketchesin the interior of the block 26. The user can also place any number ofsketches along a given axis. The 3-D sketch system interpolates betweenthose sketches to create a smooth shape. Typically, this cannot be donein physical blue foam. The 3-D sketch system also allows the user tocreate sketches with holes in them and to use initial shapes other thana block. The other advantages include having easy undo and redofunctions and having a virtual object 26 as a digital version of themodel that can be passed on to CAD/CAM programs, rapid prototypingmachines, etc.

FIG. 28A illustrates an initial block of virtual material 701 overlaidby construction planes 700 a, 700 b, 700 c, 700 d, aligned along an axis703 and a sketch 702 on the top surface of construction plane 700 a. Inone embodiment, the starting block of virtual material 701 is a virtualobject 26, as described previously. In the embodiment shown in FIG. 28A,the construction planes 700 occupy the same space as the starting blockof virtual material 701; that is, the planes 700 overlay the same spaceas the starting block 701. In other embodiments, the construction planes700 occupy a larger or smaller space than the virtual object 26. Inanother embodiment, only a starting construction plane 700 a and anending construction plane 700 d connected by an axis 703 need be used.In another embodiment, an ending construction plane 700 d is notrequired. In a further embodiment, all that is required is a startingconstruction plane 700 a with a sketch 702 on the plane 700 a.

If several construction planes are provided along one axis 703, theyform a stack 718 of construction planes 700. In one embodiment, theconstruction planes 700 are aligned along an axis 703. In oneembodiment, the axis 703 may be positioned at any orientation relativeto the initial block of material 701. In one embodiment, the user drawsthe sketch 702 on the top construction plane 700 a using the hapticinterface device 10. In another embodiment, the user places a previouslycreated sketch or 2-D design on the construction plane 700 a. There isno requirement that the sketch 702 be placed on a top or outermostsketch plane 700 a, and the sketch 702 may be placed on any constructionplane 700 a, 700 b, 700 c, 700 d. There is no requirement that thevirtual object 26 be a box-like or rectilinear shape, as shown in FIG.28A. The starting block of material of the virtual object 26 can be anysuitable shape that the user finds useful.

FIG. 28B illustrates a cutaway or resulting block of material 704 basedon the sketch 702. The user invokes a cutting or “cut outside” operationbased on the sketch 702 to produce the cut block of material 704. FIG.28C illustrates a holed block of material 706 based on the sketch 702.In FIG. 28C, the user invokes a “cut inside” operation based on thesketch 702 to remove material to create a hole 708 in the resultingholed block of material 706. The result is not required to be a hole708. For example, if the sketch extends to the edge of the constructionplane 700A, then the result of the cut outside operation could be asubstantially U-shaped or otherwise open cavity.

FIG. 29A illustrates the cut block 704 of FIG. 28B overlaid with a blockof potential material of the virtual environment 701, which is overlaidwith a starting construction plane 710 a, and ending construction plane710 b, and a sketch 712 on starting plane 710 a. The user implements acreate operation to create the union block of material 714 shown in FIG.29B. The union block 714 includes horizontal portions 716 a and 716 band a vertical portions 704 a and 704 b. The vertical portions 704 a and704 b correspond to the cut block of material 704 shown in FIG. 28B. Thehorizontal portions 716 a and 716 b are determined by the sketch plane712 on the starting construction plane 710 a in FIG. 31A. The user canrepeat the process described for FIGS. 29A and 29B for additional blocksof material 701 overlaid with additional construction planes 700 to addadditional portions to the union block 714 to create an increasinglycomplex object. In one embodiment, the user can use a cut outsideoperation to put holes 708 or cavities in a cut block 704 or union block714 to produce complex shapes with concave and convex surfaces, as wellas cavities, holes, and protrusions of various shapes.

FIG. 30 provides a flowchart of the process of making a producing amodified virtual object or cut block of material 704 from an initialvirtual object or starting block of material 701 without requiring anyforce feedback interaction. First the user selects or provides aninitial virtual object 704 (step 730). Then the user provides an axis703 that intersects the initial virtual object 704 (step 732). In oneembodiment, the axis is oriented to intersect the initial virtual object704 by default, as shown in FIG. 28A. In another embodiment, the userorients the axis 703 with a hand tool 792, as shown in FIG. 33B. Theuser then specifies or selects a starting construction plane: 700 (step734). In one embodiment, the 3-D sketch software provides one or moreconstruction planes 700 with an axis 703 that intersects the initialvirtual object 704. In one embodiment, the user does not need to specifyor use any axis 703, but may use construction planes 700 only. Inanother embodiment the user aligns a stack of construction planes 700 tooverlay the initial virtual object 701). The user then provides a twodimensional sketch 702 on the starting construction plane 700, such asplane 700 a in FIG. 28A (step 736). The user may also place the sketchon some other plane, if more than one construction plane 700 has beenselected or provided. The user may draw the sketch directly on a surfaceof the construction plane 700 using a drawing tool, such as a virtualdrawing tool 690, or create the sketch in a separate drawing program andthen move the sketch on the construction plane 700. The user theninvokes an operation to apply the sketch 702 to the initial virtualobject 701 (step 738). The 3-D sketch software then divides the initialvirtual object 701 into a first and second portion by slicing theinitial virtual object 704 based on the sketch 702 and the orientationof the axis 703 (step 738 continued). The user selects, or has selectedpreviously, one selected portion of the two portions (step 740) and thenthe 3-D software creates a modified virtual object, such as a cut blockof material 704 or a holed block of material 706, based on the selectedportion of material (step 742). The user can then request output of themodified virtual object to a data storage medium, to a visual display,over a network, to a printed representation, or a 3-D physical model.

FIG. 31 provides a flowchart showing a process of producing a modifiedvirtual object or cut block of material 704 from an initial virtualobject or starting block of material 701 in a haptic virtual environmentwhile providing force feedback to the user of the software. First theuser selects a virtual tool 28, such as a virtual drawing tool 690, touse in the haptic virtual environment (step 750). For example, thevirtual drawing tool 690 may appear as a virtual pen or pencil that theuser can use to draw a sketch 702 on a construction plane 700. Then theuser selects or provides an initial virtual object 704 (step 752). Thelocation of the user in space is determined (step 754). In oneembodiment, this location is the location of a haptic interface device10, such as a stylus or pen, that the user is holding or manipulating inreal space. Then the user or the sketch software orients an axis 703that intersects the virtual object 701 (step 756). The user then selectsa starting construction plane 700 that intersects the axis 703. (step758). In one embodiment, the sketch software provides a startingconstruction plane 700 when the user selects a sketch mode. The userthen provides a 2-D sketch 702 on the starting construction plane 700(step 760). Typically, the user draws the sketch 702 directly on asurface of the construction plane 700 using the virtual drawing tool690. In one embodiment, the software senses the position of the virtualdrawing tool 690 and provides haptic or force feedback to the user asshe draws the sketch 702. The force feedback aids the user in keepingthe virtual tool 28 on the construction plane 700 when making the sketch702. In addition, the force feedback provides feedback based on point,line, or circle drawing constraints. For example, when one drawingentity, such as a line, is close or aligned with another drawing entity,the user feels a haptic force feedback effect, which will be discussedin more detail later. The 3-D sketch software then divides the initialvirtual object 701 into a first and second portion by slicing the stackof planes 700 based on the sketch 702 (step 762). The user selects, orhas selected previously, one selected portion of the two portions (step764), and then the 3-D sketch software creates a modified virtualobject, such as a cut block of material 704 or a holed block of material706, based on the selected portion of material (step 766), as describedpreviously. The user can then request output of the modified virtualobject to a data storage medium, to a visual display, to a remotelocation over a network, to a printed representation, or a 3-D physicalmodel.

FIG. 32A illustrates an initial virtual object 701 with sketches 776,778, and 780 on construction planes 700 a, 700 c, and 700 d. FIG. 32Billustrates a modified virtual object 782 created by an interpolationprocess from the initial object 701 and sketches 776, 778, and 780.Referring to FIG. 32A, the user first places the sketch 776 on plane 700a, a different sketch 778 on plane 700 c, and another sketch 780 onplane 700 d. Then the user applies the sketches, 776, 778, 780 to theinitial virtual object 701. The 3-D sketch software then interpolatesamong the sketches to produce the modified virtual object 782, which iscomposed of top portion 784 and bottom portion 786, as shown in FIG.32B. A top portion 784 of the virtual object 782 is determined byinterpolation between sketch 776 on plane 700 a and sketch 778 on plane700 c. A bottom portion 786 of the virtual object 782 is determined byinterpolation between sketches 778 on plane 700 c and sketch 780 onplane 700 d.

FIG. 33A depicts a virtual object 26 with starting construction planes788 and 790 connected by an axis 703. The construction planes 788 and790 are placed a distance from the virtual object 26 and are notrequired to touch or be contiguous with the virtual object 26. The usercan draw a sketch 702 on the starting construction plane 788 and thenapply the sketch 702 to the virtual object 26 to produce a cut block ofmaterial 704, a holed block of material 706 or union block of material714 in a manner similar to what has been described previously. The usercan move or rotate the axis 703, so that the axis 703 intersects thevirtual environment at a different angle, as shown in FIG. 33B. In oneembodiment, the starting construction plane 788 and ending constructionplane 790 are connected in a rigid manner to the axis 703. The user canuse a grasping or hand-shaped virtual tool 792 to grasp the axis 703 androtate or move the axis 703 through the virtual space, thus causing thestarting construction plane 788 and ending construction plane 790 tomove in coordination with the axis 703.

Scene Graph and Nodes

One embodiment of the 3-D sketch approach is implemented by SensAbleTechnologies, Inc. of Cambridge, Mass., as described in the followingsections. The 3-D sketch software relies on GHOST® SDK software fromSensAble Technologies.

In one embodiment, the basic structure of the system is a GHOST SDKscene graph 840 made from three custom GHOST SDK nodes: the sketch blocknode 842, the plane stack node, referred to generally as 844, and thesketch plane node, referred to generally as 846. FIG. 36 illustrates ablock diagram of a scene graph 840, including a sketch block node 842,plane stack nodes 844 a, 844 b, and sketch plane nodes 846 a, 846 b, 846c, 846 d, 846 d one embodiment, the sketch plane node 846 corresponds tothe sketch or construction plane 700, and the plane stack node 844corresponds to the stack 718, as illustrated in FIG. 28A.

In one embodiment, the sketch block node 842 is a GHOST separator node.The sketch block object 842 has a child node a GHOST shape node thatcontains both the starter shape and the volume representations for thecurrent virtual object 26, as well as the graphic (polygonal)representations of the current virtual object 26. This child node is notshown in FIG. 36. All of the plane stacks associated with the virtualobject 26 are stored as children nodes 844 in the GHOST scene graph 840of this object. Therefore, by transforming this object, the usertransforms the virtual object 26 and all of the sketch planes relativeto the world reference frame.

In one embodiment, the plane stack node 844 is a GHOST separator node.The plane stack node 844 has as its children the sketch plane nodes 846for the stack 844. The plane stack node 844 represents the stack 718 ofconstruction planes 700 observed and manipulated by the user in thevirtual environment. Transforming the stack 718 transforms the stackaxis 703 and all of the sketch or construction plane nodes 846 in thestack node 844 relative to the sketch block node 842. The sketch planechildren nodes 846 of a plane stack node 844 are always sorted by theposition of the sketch planes 700 along the axis 703 of the plane stack718.

In one embodiment, the sketch plane node 846 is a GHOST shape node. Thisnode 846 represents a haptic rectangle or other shape that is“haptically transparent.” The sketch plane node 846 contains a bitmaprepresenting the grid and the pencil drawings in that plane 846 as wellas a representation of the ink sketch in that plane 846. By transformingthe sketch plane 846 relative to its parent stack 844 the user canchange the position of the plane 846 along the axis of the parent stack844.

As mentioned previously, there are two threads in the system—the servoloop thread running at 1000 Hz and the graphics thread running atapproximately 30 Hz. In one embodiment, the servo thread is a highpriority process and controls the force feedback that the user feelsthrough a haptic interface device 10 (reading position and outputtingforce). In another embodiment, the user uses the 3-D sketch approach tomodify virtual objects 26 without requiring any force feedback. Thegraphics thread does all of the graphic rendering and handles all thewindow events such as mouse, keyboard, and menu events. Creation andscheduling of the servo thread is handled by GHOST® SDK. In order forthe force feedback to be stable and high quality, the servo thread mustmaintain its high rate of 1000 Hz. In order to do that, many costlyoperations that would otherwise be done in the servo thread are done inthe graphics thread. The two threads have shared memory makingcommunication easy via setting of shared flags etc.

Sketching

When the user touches a sketch plane 700 using a virtual drawing tool690, the sketch plane 700 updates the pencil sketch or ink sketch 702appropriately depending on the current tool 690 being used. The tool 690may be either a pencil tool or an ink tool. The tool 690 is activated bytouching the sketch plane 700 and depressing the stylus switch on thehaptic interface device 10. The tool 690 remains active until the userlets go of the stylus switch or stops touching the plane 700.

Pencil

Pencil drawing is stored as a pixmap (pixel map) in the standard formatfor OpenGL texture maps. The grid that is drawn on the sketch plane 700is also stored in this texture map. The texture map is a map of thesurface texture including individual texture elements, or texels, thatis associated with a graphic object. Initially the entire texture map isset to the background color except those texels that are on grid lineswhich are set to the grid line color. In one embodiment, the drawing anderasing of the texture map is updated at the servo rate (1000 Hz),although the updated texture map is sent to the OpenGL renderer at thegraphics rate (30 Hz).

If the sketch plane 700 is touched while the stylus switch of the hapticinterface device 10 is depressed while the virtual drawing tool 690 inpencil mode, the pencil draws. Drawing with the pencil involves changingthe color of the texel currently being touched by the user to the pencilcolor. In one embodiment, the texture map is updated this way at theservo rate, although the updated texture map is sent to the OpenGLrenderer at graphics rate (30 Hz).

If the sketch plane 700 is touched while the stylus switch of the hapticinterface device 10 is depressed, the drawing tool 690 is in erasermode, and the user is specifying they'd like to erase pencil marks, thenine texels forming a square around where the tool 690 is touching areerased. Erasing a texel involves setting it to either the background orthe grid line color as needed.

Ink

Ink drawings are stored as geometric primitives (line segments, circles,arcs, freehand curves, and boxes). They are not rendered into thetexture map as with pencil drawings. They are rendered using OpenGL linedrawing routines overlapping the sketch plane 700 they belong to. In oneembodiment, ink geometry is created and modified in the servo thread,but new geometry is added to the sketch 702 (the list of ink geometryfor the plane 700) in the graphics thread. The servo thread signals thegraphics thread that new geometry is to be added and the graphics threadadds it the next time it goes through the rendering loop.

Drawing

Ink drawing creates and updates the geometric primitives at the servorate although they are redrawn at the graphics rate. During all drawingwith the drawing tool 690 set to an ink mode, the user is constrained tothe sketch plane 700 being drawn on. The constraint begins once thedrawing tool 690 touches the plane 700 and the user depress the stylusswitch on the haptic interface device 10 which activates the tool 690.The ink drawing tool 690 remains constrained to the plane 700 until theuser lets go of the stylus switch at which point the tool 690 becomesinactive.

a) Line Segment

Line segments (also referred to as lines herein) are represented bytheir start point and endpoint. The user may use the drawing tool 690set to a line drawing mode, which is also referred to as a line tool.When using the line tool, if the user touches the plane 700 with thedrawing tool 690 while depressing the stylus switch on the hapticinterface device 10, the point of first contact is stored. The startpoint of the line is set to this point. The end point of the lineinteractively follows the location (haptic interface location 98) of thehaptic interface device 10 as the user moves the line tool around theplane 700. Once the user lets go of the stylus switch, the end point ofthe line becomes fixed at its last location.

b) Circle and Arc

Circles are represented as arcs by a center, radius, start angle andspan angle. The user may use a drawing tool 690 set to a circle mode,which is also referred to as a circle tool. When using the circle tool,if the user touches the plane 700 with the stylus switch of the hapticinterface device 10 depressed, the point of first contact is stored. Thecenter of the circle is set to this point. The radius of the circleinteractively changes as the user moves the circle tool in the plane 700so that the circle passes through the point being touched by the user.Once the user lets go of the stylus switch on the haptic interfacedevice 10, the radius becomes fixed at its last value. Initially thestart angle is 0 degrees and the span angle is 360 degrees representinga complete circle. These angle values change if the circle is trimmed.

c) Box

A box is represented by two points: the top left corner and the bottomright corner. The user uses a drawing tool 690 set to a box mode, alsoreferred to as a box tool. When using the box tool, if the user touchesthe plane 700 with the drawing tool 690 while depressing the stylusswitch of the haptic interface device 10, the point of first contact isstored. The top left corner of the rectangle is set to this point. Thebottom right corner of the rectangle interactively follows the location(haptic interface location 98) of the haptic interface device 10 as theuser moves the box tool around the plane 700. Once the user lets go ofthe stylus switch, the bottom right corner of the rectangle becomesfixed at its last location. Once a box has been created, it is convertedinto four lines representing the sides of the box.

d) Freehand Curve

A freehand curve is represented by an ordered list of points. The curveis approximated by the line segments connecting the points. When usingthe drawing tool 690 set to a freehand curve mode, if the user touchesthe plane 700 with the drawing tool 690 and depresses the stylus switchon the haptic interface device 10, the point of contact is stored as thestart point for the curve. As the user moves the drawing tool 690 in theplane 700, the current location (haptic interface location 98) of thehaptic interface device 10 is added to the list of points for the curveeach time the stylus is depressed and released (clicked). After each newpoint is added to curve, a spline curve is fit through the curve points.Any prior temporary curve displays are erased and the resulting smoothcurve is displayed to the user as a temporary curve display reflectingthe current state of the curve placement operation. After the usersignifies that he is finished providing curve points (by typing theletter ‘E’ in one embodiment), the temporary curve display is erased andthe ink mode sketch freehand curve is obtained by a spline curve fitthrough all the user provided curve points.

Snap-To Constraints

Snap-to constraints are implemented using a constraint effect. Snap-toconstraints used on a sketch plane 700 in a 3 D sketch mode should bedistinguished from geometric constraints used in a 3 D sculpting mode.The snap-to constraint effect can constrain the user to either a point,a line segment, a circle, an arc, a box, or a freehand curve. Todetermine which if any type of constraint to use, the 3 D sketchsoftware checks the current location of the user (haptic interfacelocation 98) and the current geometry being created against the othergeometry that is already in the sketch. If it finds a constraintcondition, such as a parallel line or tangent circle, it sets theconstraint effect appropriately. See FIGS. 34A-34D and 35, which arediscussed below. Once a constraint is set it stays until the user movesout of the constraint condition. Since the constraint forces the user toremain in the constraint, she must effectively exceed a certain forcethreshold to leave the constraint.

When not drawing (stylus switch on the haptic interface device 10 is up)point constraints are set for the endpoints and midpoints of lines, thecenters of circles, the center and endpoints of arc, the corner pointsof boxes and the start and end points of freehand curves. The user isconstrained to a point as soon they are within a predetermined “snapradius” 806 of the point. See FIG. 34A. The constraint ends once theymove out of the snap radius 806. Each point that the user can beconstrained to has a circle with a radius equal to the snap radius 806drawn around it.

When drawing lines all of the point constraints discussed above areactive. In addition, if the current line being drawn is parallel toanother line in the sketch, the user is constrained to move only alongthe current line until she pushes hard enough to move out of theconstraint condition. The same is applied for lines in the sketchperpendicular to the line being drawn and for circles tangent to thecurrent line being drawn.

When drawing circles all of the point constraints above are active. Inaddition if the circle being drawn is tangent to a line or circle in thesketch, the user is constrained to move only in the circle being drawn.

FIGS. 34A-34D and 35 illustrate these constraints in more detail. FIG.34A depicts a virtual object 26 with a point constraint on aconstruction plane 800. On the sketch plane 800 is an existing line 802that the user has previously drawn. The existing line 802 has anendpoint 804 and a snap-to area or radius 806. In one embodiment, thesnap-to area 806 is a circle centered on the endpoint 804. In otherembodiments, the snap-to area has other shapes than a circle or may havea 3 D shape such as a sphere or cylinder as will be discussed later. Theuser is drawing a new or current line 808. If the end 810 of the newline 808 touches the snap-to area 806, then the end 810 of the new line808 snaps to and connects with the endpoint 804 of the existing line802. In one embodiment, the user feels a force feedback reflecting thesnap-to effect of the end 810 of the new line 808 being attracted to andsnapping to the endpoint 804. The user uses a virtual tool 28, such as avirtual drawing tool 690, to draw or manipulate the new line 808 with ahaptic interface device 10.

FIG. 34B depicts a virtual object 26 with a sketch plane 800 showing anexisting line 802 and additional existing lines 812 and 814. The user isdrawing a new line 816. Whenever this line 816 is parallel to theexisting line 802, within a predetermined limit, then the new line 816snaps to a position parallel to the existing line 802. In oneembodiment, the existing line 802 and the new line 816 are highlightedwhen the new line 816 is in a parallel orientation to the existing line802. The lines 802, 816 may be highlighted by a different color,different gray tone, different thickness, dotted manner, dashed manner,or other approach typically used to highlight lines. In one embodiment,the user feels a force feedback effect when the snap-to effect occurs asthe user moves the new line 816 in different orientations using avirtual tool 28, such as a virtual drawing tool 690, to move the newline 816 into different positions. The user may also select an existingline, such as 814, and move it into a different orientation until line814 becomes parallel to existing line 802.

FIG. 34C depicts a virtual object 26 with a sketch plane 800 showing anexisting line 802 and additional line 814. The user is drawing a newline 818.

Whenever this line is perpendicular to the existing line 802, within apredetermined limit, then the new line 818 snaps to a positionperpendicular to existing line 802. In one embodiment, the user feels aforce feedback effect when the snap-to effect occurs, as the user drawsor manipulates the new line 818 with a virtual tool 28, such as avirtual drawing tool 690.

FIG. 34D depicts a virtual object 26 with a sketch plane 800 showing anexisting line 802. The user is drawing a new circle 820. When the circlecomes close to the existing line 802, the circle 820 snaps to theexisting line 802, which thus becomes a tangent to the circle 820. Inone embodiment, the user feels a force feedback effect when the snap-toeffect occurs, as the user draws or manipulates the circle 820 with avirtual tool 28. Conversely, if the user is drawing a new line and itcomes close to the circle 820, the line may snap to the circle andbecome a tangent to the circle 820 (not shown in FIG. 34D).

FIG. 35 depicts a snap-to area for a point 824, for one embodiment ofthe invention shown. The snap-to area is implemented as a 3-Dcylindrical volume 828 centered on the point 824 and extending above andbelow the sketch plane 800. This cylindrical configuration 828 providesfor a snap-to effect if the virtual tool 26 is somewhat above or belowthe surface of the sketch plane 800. Thus if the user is approaching thepoint 824 with a virtual tool 28 from above the plane 800, the userfeels an attractive feedback force to the point 824 when the virtualtool 28 enters the cylindrical volume 828.

Editing (Pliers)

The pliers tool lets the user edit previously created geometry. The useruses a drawing tool 690 set to a pliers mode. When in pliers mode, ifthe sketch plane 700 is touched; the list of ink geometry in the plane700 is searched to see if any of it is being touched. If no geometry istouched, nothing happens. If a piece of geometry is touched, it ishighlighted graphically by drawing it in another color or using a dashedline, a heavier line, or some other highlighting effect. If geometry istouched and the stylus switch of the haptic interface device 10 isdepressed, then the geometry is edited using a method specific to thattype of geometry. All modification to geometry other than the freehandcurve is done at the servo rate.

a) Line

A line can only be touched by the pliers if one of the lines endpointsis touched. If the user touches one of the endpoints and depresses thestylus switch of the haptic interface device 10, that endpoint tracksthe location of the haptic interface device 10 until the stylus switchis released. Once the stylus switch is released the picked endpoint ofthe line is set to the last location (haptic interface location 98) ofthe haptic interface device 10.

b) Circle or Arc

If the user touches a circle or arc using the pliers and depresses thestylus switch of the haptic interface device 10, the radius of thecircle or arc interactively changes as the user moves the pliers in theplane 700 so that the circle or arc passes through the point beingtouched by the pliers. Once the user lets go of the stylus switch theradius becomes fixed at its last value.

c) Box

Once the user draws a box in the sketch plane, it is converted to fourlines representing the sides of the box. It is not edited as a box. Eachline may be edited individually.

d) Freehand Curve

If the user touches a freehand curve using the pliers and depresses thestylus switch of the haptic interface device 10, the curve is modifiedby recalculating a spline definition that is identical to the majorityof the original curve, but changes a local area around the point on thecurve specified by the user so that the modified curve passesdynamically through the changing location of the haptic interface device10. Then the stylus switch is released, the original curve is replacedwith the last seen dynamic curve definition. The methods for dynamicallymodifying spline curves are well known to those skilled in the art ofB-spline curves.

Erasing (Trimming)

The eraser has two functions. The user uses a drawing tool 690 in aneraser mode, also referred to as an eraser tool. If the user touches thesketch plane 700 with the eraser tool, depresses the stylus switch ofthe haptic interface device 10 and chooses one of the two functions (inone embodiment by applying a set amount of force), then pencil tool 690marks where the eraser touches are erased (see above). The eraser alsocan function to erase ink geometry. If the user depresses the stylusswitch on the haptic interface device 10 while touching ink geometry,then the geometry being touched is trimmed or deleted.

FIG. 37 provides a pictorial view of an open curve 850 to be trimmed.The open curve intersects a first line 852, a second line 854, a thirdline 856, and circle 858. The points of intersections between the opencurve 850 and the other pieces of geometry, 852, 854, 856, 858 aremarked by intersection points, T2, T3, T5, and T6. An eraser tooltouches the plane 700 and the open curve 850 at point T4.

The trimming algorithm works as follows: First the geometry to betrimmed (trim geometry) is determined. In general the trimming is doneby adding a new piece of geometry that represents the trimmed piece, anddeleting the original piece of geometry (850). In this case, the opencurve 850 is intersected with the rest of the geometry 852, 854, 856,858 in the sketch and a list of intersection points is created asindicated by points T2, T3, T5, and T6. If there are no intersectionpoints, then the trim geometry is simply deleted, otherwise it istrimmed. Let P_(start) (T1) and P_(end) (T7) be the start and end pointsof the trim geometry. The two closest intersection points T3 and T5surrounding the point at which the geometry was touched T4 are found. IfT3 is found then a new piece of geometry is added to the sketch. Thisnew piece of geometry has start point T1 and endpoint T3. Similarly, ifT5 is found, then a new piece of geometry is added to the sketch. Thisnew piece of geometry has start point T5 and endpoint T7. The originaltrim geometry 850 is deleted, and the new pieces of geometry from T1 toT3 and from T5 to T7 are retained. Line segments, arcs, boxes, andfreehand curves are trimmed according to this algorithm.

In the special case of a closed circle that has no start or end point,the process is similar. If there is only one intersection point then thewhole circle is deleted. Otherwise the closest points to the touchedpoint are found as above, the original circle is deleted and a new arcis added having the same center and radius as the circle but with endpoints such that the arc does not pass through the touched point. Inaddition to the above modifications to sketch entities, the user alsohas control to move, rotate, scale, and delete individual or groups ofthese entities. In one embodiment, these modifications include theconstraints mentioned above. In another embodiment, these modificationsare enhanced through haptic feedback.

In one embodiment, the 3-D sketch software does not support trimming offreehand curves. They are always deleted in their entirety. In anotherembodiment, the routines to calculate the intersection of a line with afreehand curve and the intersection of an arc with a freehand curve. Inanother embodiment, the trimming process is made faster by a spatialpartition method (BSP tree or quadtree) on the curve points.

Selection of the trim object is done in the servo thread. The servothread then signals the graphics thread that trimming must be done andtells it which geometry to trim. The trimming itself is done in thegraphics thread since it can be time consuming.

Mirroring

The mirroring user interface in 3 D sketch mode works as follows. Theuser can enable mirror mode either by selecting “enable mirror” or“place mirror.” If the user selects “enable mirror,” then the mirrorline is either the last mirror line set for this plane 700 or thedefault mirror line if no previous mirror was created. If the userselects “place mirror” then the user draws the new mirror line in theplane 700—this also enables mirroring. Once mirroring is enabled, theuser then selects the entities to mirror by touching them and depressingthe stylus switch on the haptic interface device 10. Selected entitiesare highlighted in another color or some other highlighting mechanism.If the user touches an already selected entity and depresses the stylusswitch of the haptic interface device 10, the entity is unselected andunhighlighted. When the user selects “execute mirror” all of thecurrently highlighted entities are mirrored using the current mirrorline. Mirroring remains enabled until another tool is selected. In oneembodiment, only ink entities are mirrored.

There are three parts to mirroring: placing the mirror line, selectingentities to mirror and executing the mirror. In one embodiment, placingthe mirror line and selecting entities to mirror are implemented asseparate tools. The user uses the tool 690 in a place mirror mode or aselect entities mode, also referred to as a place mirror tool or aselect entities tool.

The place mirror tool is the same as the line drawing tool except thatthe line drawn is not added to the sketch, and it is drawn in adifferent color. Otherwise placing the mirror line is exactly likedrawing a line with the ink line tool and even the haptic constraintsare enabled. The user does not have to explicitly place the mirror line.Every plane 700 has a default mirror line. Once the mirror line has beenplaced in a plane 700, that mirror line is stored, and the next timemirroring is enabled, it appears as the default mirror line.

The select entities tool searches the current sketch every time the userdepresses the stylus switch of the haptic interface device 10 to see ifany geometry is touched. A list of selected geometry is maintained. Whengeometry is touched, it is either added or removed from this list.

When “execute mirror” is selected each of the entities in the mirrorlist is mirrored by the current mirror line and the mirrored version isadded to the sketch. Lines are mirrored by mirroring their endpoints.Circles are mirrored by mirroring their centers. Arcs are mirrored bymirroring their centers and then mirroring the start point of the arcand using that to calculate a new start angle (Similarly for end pointand span angle). Freehand curves are mirrored by mirroring each of thepoints in the curve.

Selecting entities and adding/removing them from the mirror list is donein the servo thread. Executing the mirroring is done in the graphicsthread.

Sketch Planes

Haptics

Haptically the sketch or construction plane 700 is represented as arectangular shape in three dimensions or an infinitely thin rectangularbox. The width and height of the rectangle are stored. It is defined inits own coordinate reference frame so that the normal of the plane 700is the positive z-axis and the center of the rectangle is at the origin.The sketch plane 700 is implemented as a GHOST® SDK gstShape node;therefore it uses the basic GHOST SDK haptic model. For each iterationof the haptic rendering process, a new SCP 226 is calculated based onthe last SCP 224 (SCP_(last)) and the current position (P) or hapticinterface location 98 of the haptic interface device 10, implemented inone implementation with the PHANToM® device from SensAble Technologies.First both of these points, SCP and P, are transformed into the planescoordinate reference frame. This means that a point is on the plane 700if its z coordinate is zero. If both SCP_(last) and P are on the same ofthe plane there is no collision and so SCP=P. If they are opposite sidesof the plane 700 then P is projected onto the plane 700. Call thisP_(proj). If P_(proj) is outside of the rectangle then there is nocollision, so SCP=P; otherwise, there is a collision and SCP=P_(proj).

When the user is not holding down a switch on the haptic interfacedevice 10, the plane 700 is “haptically transparent”. Haptictransparency means that, if the user pushes hard enough on the plane 700with the stylus, then the virtual drawing tool 690 pops through theplane 700. This allows the user to interact with planes 700 and thevirtual object that may be obscured by other planes 700. This isimplemented by checking the distance between P and P_(proj) in worldcoordinates. If this distance times the spring stiffness of the sketchplane 700 exceeds a force threshold, then the virtual drawing tool 690has popped through. In this case no collision is reported, and SCP=P. Ifthe threshold is not exceeded a collision is reported and SCP=P_(proj)as above.

If the user depresses the stylus switch, then the sketch plane 700 isnot set to be haptically transparent. This is done so that the user doesnot fall through the plane 700 while drawing or editing with the virtualdrawing tool 690. In addition while the stylus switch is depressed, thehaptic interface device 10 is constrained to the plane 700. Once theuser touches the plane 700 while the stylus switch is down, theconstrainToPlane flag is set. This flag is cleared when the stylusswitch is released. As long as this flag is set, the sketch plane 700always reports a collision and sets SCP=P′_(proj) where P′_(proj) isP_(proj) moved to the appropriate edge of the rectangle.

If the user depresses the stylus switch and the plane 700 being touchedis not the “touched” plane and some other plane is the “touched” plane(see selection below), it never reports a collision. This prevents theuser from feeling other planes 700 through the force feedback receivedthrough the haptic interface device 10 that intersect the current planewhile drawing.

All of the haptic calculations are performed by the haptic renderingprocess.

Selecting

There are two important concepts in the way selection is implementedfor: the touched plane and the working plane (or last touched plane).

There can be only one touched plane at a time. If no plane is currentlyin contact with the haptic interface device 10, then there is no touchedplane. In this case the first plane to come in contact with the hapticinterface device 10 or virtual drawing tool 690 becomes the touchedplane, and no other plane may become the touched plane until the currenttouched plane is no longer in contact with the virtual drawing tool 690.If the user depresses the stylus switch on the haptic interface device10, only the touched plane will ever report a collision with the virtualdrawing tool 690. This prevents the user from feeling other planes thatintersect the current plane while drawing.

The working plane is the last plane that was set to be the touchedplane. When no plane is currently in contact with the virtual drawingtool 690, there can be no touched plane but there can be a workingplane. The working plane is highlighted and counts as the selected planefor operations such as clear plane, delete plane, cut plane, etc.

Both the touched and working plane are set in the haptic renderingprocess 16.

Moving

Sketch planes 700 are moved using the hand tool. In one embodiment, thehand tool is one end 694 of the virtual drawing tool 690. When the usertouches a sketch plane 700 with the hand tool and the stylus switch isdepressed, the plane 700 may be moved. The plane 700 is moved bytransforming it along the z-axis of the parent plane stack 718 in thedirection the haptic interface device 10 causing the virtual drawingtool 690 to move.

The force felt from moving a plane 700 and the new position of the plane700 as it moves are calculated using a 3-D friction model. The model isthat of dragging a massless block with the haptic interface device 10.When the user first touches the plane 700 with the stylus switch down,the position of the block is set to the position of the plane 700. Whenthe haptic interface device 10 moves, a spring force is calculatedbetween the current location or haptic interface location 98 of thehaptic interface device 10 and the position of the block using Hooke'sLaw. If the spring force is below a predetermined threshold, then theblock does not move and the spring force is sent to the haptic interfacedevice 10. If the spring force exceeds the threshold, then the block ismoved to a point some fixed distance from the haptic interface location98 of the haptic interface device 10 along the line segment from thehaptic interface device 10 to the block position. In this case, theforce sent to the haptic interface device 10 has the same direction asthe spring force but has the magnitude of the fixed force threshold. Asthe user moves the haptic interface device 10, the position of the plane700 is set to that of the block in the simulation. The force effect thatthe user feels in the haptic interface device 10 gives the user theimpression that she is dragging the plane 700 along. It also filters outthe jitter in the human hand. In addition, the block is constrained tomove only parallel to the axis 703 of the parent plane stack 718, andthe haptic interface device 10 is also constrained to only move parallelto that axis.

All plane moving is done in the haptic rendering process 16. In oneembodiment, plane moving this is done using a GHOST SDK manipulator nodethat allows the user to transform other nodes and send force to thehaptic interface device 10.

Adding

In one embodiment, adding new sketch planes 700, means adding a newchild sketch plane node 846 to the appropriate plane stack node 844.Adding a new child 846 into the scene graph 840 from the graphicsprocess 22 while the haptic rendering process 16 is running can createsynchronization problems.

Removing the plane 846 in the haptic rendering process 16 while thegraphics process 22 is running can cause the same problems. Creating anew plane 846 is a time consuming process that cannot be done in thehaptic rendering process 16. The 3-D sketch software therefore does thisin two parts. First the new plane 846 is created in the graphics process22 and all of the parameters (width, height, stiffness, etc.) are set tothose of the plane stack 844 that new plane 846 will be added to. Thenthe haptic rendering process 16 is signaled that a plane 846 must beadded. The graphics process 22 then waits for the haptic renderingprocess 16 to add the plane 846 to the appropriate stack before tryingto render the scene again. The haptic rendering process 16 adds theplane 846 and then signals the graphics process 22 it has completed.

Deleting

In one embodiment, deleting a sketch plane 700 consists of removing itas a child sketch plane node 846 from its parent plane stack 844. Thiscannot be done in the graphics process 22 so the haptic renderingprocess 16 is signaled and it removes the plane 846. The graphicsprocess 22 waits for it to be removed from its parent 844 and thendeallocates the memory.

Clearing

The user may clear either the ink, the pencil or both. Clearing a plane700 consists of removing the list of ink entities (the sketch 702)and/or resetting the texels in the texture map to the clear color. Thisis done in the graphics process 22; however, a flag is set while theclear is happening so that the haptic rendering process 16 does not tryto manipulate the sketch 702 or the texture map during a clearoperation.

Cut and Paste

In one embodiment, only entire sketch planes 700 may be copied, cut, andpasted. To copy a sketch plane 700 a copy of the sketch 702 and thetexture map are made and stored in the clipboard. A cut operation isequivalent to a copy followed by a clear. To paste the plane 700 to bepasted to has its sketch 702 set to a copy of the sketch in clipboardand its texture map set to a copy of the texture map in the clipboard.When the texture map in the clipboard is copied into the new plane 700,it is centered in the new plane 700 and clipped to the boundaries of thenew plane 700.

Virtual Object Generation

The sketch planes 700 are used to generate a virtual object 26 or tomodify an existing virtual object 26 or initial block of material 701.In one embodiment, the metaphor for generating a modified virtual object26 is based on blue foam and a hot wire. The approach is to start withsome initial shape in blue foam (starter shape) and then use thesketches to guide the hotwire to cut the piece of blue foam. In oneembodiment of the 3-D sketch software, each plane stack 718 correspondsto one cut with the hotwire. The starter shape 701 is first cut based onthe plane stack 718 along the x-axis, then the resulting shape is cutbased on the sketch planes 700 along the y-axis and finally the resultis cut by the sketch planes 700 on the z-axis to create the finalvirtual object 26.

With blue foam in the real world, the user can only cut one profile at atime, but in one embodiment of the 3-D sketch system, each plane stack718 defines a single cut made up from multiple aligned sketch planes700. The hotwire can be viewed as stretching and bending so that itpasses through all of the sketches 702 on each of the planes 700 in thestack 718. In one embodiment, the 3-D sketch software interpolatesbetween the sketches, such as 776, 778, and 780 in FIGS. 32A and 32B, tocreate a smooth shape that fits every sketch as if the hotwire bends andstretches between the different sketch planes 700. Each sketch 702 maybe a different geometric shape, such as a square, circle, or irregularlyshaped closed loop, and the interpolation technique provides for asubstantially smooth surface connecting between each sketch 702.

Virtual Object Representation Using Voxels

In one embodiment, the virtual object 26 is represented as a volumeusing a voxel grid representation. Each voxel has a density valueassociated with it. The density values are interpolated between voxelgrid locations using trilinear interpolation to create a continuousfield as discussed previously. The virtual surface 25 of the virtualobject 26 is the isosurface 86 of this field at a set value. In oneembodiment, the density values range from 0 to 255 and the isosurface 86used is 128. An algorithm is used to find a polygonal representation ofthe isosurface 86 which is used to represent it graphically. Thealgorithm may be any method commonly used to generate a triangular,polygonal, or mesh surface from a voxel or point representation of ageometric solid. Haptic representation of the isosurface 86 is doneusing the same methods as discussed previously for the 3-D sculptingsystem.

Generation of Distance Fields

In one embodiment, the 2-D distance fields used in the representationstore the signed distance to a set of closed outlines.

FIG. 38 provides a pictorial representation of a sketch plane 700showing a grid with a closed outline 870, vertical grid lines 872 andhorizontal grid lines 874. Points on the closed outline 870, such as G3have a value of 0. Points within the closed outline, such as G1 and G2,have a positive value, such as 2½ (1.414 . . . ) for point G1 and 1 forG2. In other embodiments, other positive values may be used for pointsG1 and G2. Points outside of the closed outline, such as G4 and G5, havea negative value, such as negative 1 for point G4 and negative 2½ forpoint G5. In other embodiments, other negative values may be used forpoints G4 and G5. The points shown, G1 through G5, are shown forrepresentative purposes and other points on the sketch plane 700 mayalso be assigned positive or negative values based on whether the pointsare inside or outside of the closed outline 870. In one embodiment, thevalues are not required to be integer values.

A point is defined to be inside the outline 870 if the line segment fromthat point to the edge of the field intersects outline 870 an odd numberof times. The field is represented by a 2-D evenly spaced grid, as shownby the gridlines 872 and 874 in FIG. 38. Each grid location has a scalarvalue for the distance. A continuous field is defined by using bilinearinterpolation.

To create such a distance field from the sketch 702 in the sketch plane700 takes two steps. First, a set of closed loops (outlines) 870 isfound in the sketch 702 then the closed loops 870 are used to generatethe distance field.

1. Finding Closed Loops

To find a single closed loop 870 in the sketch 702 one piece of geometryG_(start) in the sketch 702 is selected at random and added to the listof geometry in the proposed closed loop. The rest of the geometry in thesketch 702 is searched for another piece of geometry G_(next) such thatthe endpoint of G_(start) matches either the startpoint or the endpointof G_(next). If G_(next) cannot be found then the method fails. IfG_(next) is found then it is added to the list of geometry in theprospective closed loop and the method is repeated withG_(start)=G_(next) until the endpoint of G_(next) matches the startpointof the original G_(start) at which point a closed loop 870 has beenfound.

To find the set of closed loops 870 in a sketch 702 a single loop 870 isfound, the geometry in the loop is removed from the sketch 702 and themethod is repeated on the remaining geometry until there is no geometryleft or no additional closed loop 870 cannot be found from the remaininggeometry. If no closed loops are found, an error situation is reportedto the user and she has the opportunity to correct the sketch.

2. Converting Loops to a Distance Field

The distance field value is computed in three stages. First the sign ofthe distance is computed at each point by deriving a bitmap showingobject inclusion for each pixel value. Then absolute (unsigned) floatingpoint distance is found. These values are combined into a signeddistance, which is then scaled to the 0 to 255 byte values stored forthe virtual object 26.

Algorithms for determining the interior and exterior of a set of closedloops in terms of a bitmask are well known to those skilled in the art.The typical approach is to use a winding rule algorithm to determine inthe inside/outside state for each pixel in the bitmask. Theinside/outside information is then readily converted to a sign of thedistance for each pixel in the bitmask.

Next the method determines the unsigned floating point distance value.For each segment in each loop 870, the method identifies the shortestdistance from the pixel to that segment. This nearest point may layalong the line segment, or at either end point. The minimum of all theseper-segment distances gives the global minimum distance from the pixelto the outline set.

Multiplying the sign and value for each pixel gives a signed distancefrom each pixel to the outline set. Since the volume values are unsignedbytes, these floating point values must then be scaled to fit the range0-255. This is done by linear scaling and clamping.

Generation of Single Stack Cut Volume

In one embodiment, the cut virtual object 704 for a stack axis 703 isgenerated by creating slices of it along its z-axis. In otherembodiments, the x-axis or y-axis is used. In other embodiments,arbitrarily oriented axes are used. The z-axis of the cut virtual object704 is set (via a coordinate system transformation) to coincide with theaxis 703 of the plane stack 718. The slices are created either directlyfrom the distance field of a sketch 702 or by interpolating between thedistance fields of two different sketches 702. To create a virtualobject 26 slice from a distance field is straightforward as long as thedimensions of the grid representing the distance field match thedimensions of the grid representing the volume slice. In one embodiment,all that needs to be done is to scale the values at the grid locationsof the distance field into the appropriate range for the voxels in thevirtual object 26.

First the distance fields generated from the sketches 702 in this stackare sorted by the offset along the z-axis of the plane stack 718 of theassociated sketch plane 700. If there are no sketches 702 in this planestack 718 then the cut virtual object 704 is “full,” In one embodiment,every voxel is set to its maximum value. Otherwise the first distancefield is converted into a virtual object 26 slice and copied into thevirtual object 26 for each slice with a z coordinate less than or equalto that of the distance fields sketch plane 700. Then the last distancefield is converted into a virtual object 26 slice and copied into thevirtual object 26 for each slice with a z coordinate greater than orequal to that of the distance fields sketch plane 700. Finally, for eachconsecutive pair of distance fields d1 and d2 with z coordinates z1 andz2 respectively the slices of the cut virtual object 704 with zcoordinate such that z1<z<z2 are set by linearly interpolating betweenthe two distance fields. Specifically the slice with z coordinate z willbe the linear interpolation of d1 and d2 with weights (1−t) and trespectively where t=(z−z1)/(z2−z1). Linear interpolation of thedistance fields is done by linearly interpolating the correspondingscalar at the grid locations. Planes 700 with no sketch 702 on them areignored during virtual object 26 generation.

FIG. 39 is a pictorial representation showing the interpolation processbetween a point V₀ on a sketch plane 880 and a point V₁ on a sketchplane 882 resulting in a new point V_(N), with an interpolated valuedderived from the values of V₀ and V₁, on an intermediate plane 884. D₀represents the distance from sketch plane 880 to intermediate plane 884,and D₁ represents the distance from sketch plane 882 to intermediateplane 884. In one embodiment, the value of V_(N) is determined by$V_{N} = \frac{{D_{0}*V_{1}} + {D_{1}*V_{0}}}{D_{0} + D_{1}}$

Typically, V₀ and V₁ represent the same point in the grid on each sketchplane 880, 882. In one embodiment the values of V₀ and V₁ represent thevalues on the density fields on each sketch plane 880, 882, as describedabove, that is, positive values if inside a closed loop 870 and negativevalues if outside of a closed loop 870. Thus if V_(N) has a positivevalue, it is within a closed loop 870 on plane 884, and if V_(N) has anegative value, it is outside of a close loop 870 on plane 884. If V_(N)has a value of zero, then it is on a closed loop 870 in plane 884.

Combining a Starter Shape and a Cut Virtual Object into Final ModifiedVirtual Object

The starter shape 701 and cut virtual objects, such as 704, arebasically combined by boolean intersection. For example, see FIGS. 29Aand 29B. Since the scalar values of the virtual objects 26 are notlimited to boolean values, however, the method uses a minimum operatorinstead of boolean AND. The process works as follows: First the currentvirtual object 26 is reset to that of the starter shape 701. Next eachcut virtual object, such as 704, is intersected with the starter shape701. For each grid location in the current volume V_(m) the method findsthe corresponding point in the local coordinate system of the cut volumeV_(c). Trilinear interpolation is used to find the scalar value of thecut volume at V_(c) and finally V_(m) is set to the minimum of the twovalues; that is, val(V_(m))=min(val(V_(m)), val(V_(c))). This is donefor each of the cut virtual objects 704 and the final result is what isleft in the current virtual object 26.

Virtual Object Manipulation

In one embodiment, there are three ways to manipulate (rotate, translateand scale) the virtual object 26 and the sketch planes 700, based on avirtual hand tool, virtual object 26 manipulation with the mouse, andvirtual object 26 manipulation with the keyboard, as discussed below.

In other embodiments, the axis of the plane stack is not confined to bea straight line (allowed to be a 3-D space curve). In other embodiments,any number of axes can be combined into a single user operation.

After the modified virtual object is produced, further modifications arepossible. Further modifications include another application of the 3-Dsketch operation, a mirroring operation, sculpting operations, change oforientation operations, or other operations applicable to the virtualobject representation chosen.

Virtual Object Manipulation with Mouse

The mouse can be used to rotate, translate and scale the virtual object26 and the sketch planes 700. When the user holds down the left buttonand the mouse moves, the virtual object 26 is rotated about the x and yaxes. The difference in the last mouse location and the current mouselocation is calculated. The virtual object 26 is rotated by someconstant times the x component of the difference in mouse locationsabout the y-axis, and then rotated by the y component of the differencein mouse locations about the x-axis. When the user depresses the CONTROLkey while simultaneously holding down the left mouse button, the virtualobject 26 is rotated about the z axis by the x component of thedifference in locations. When the user holds down the right mousebutton, the virtual object 26 is translated in the x and z axes. Whenthe user depresses the CONTROL key while simultaneously holding down theright mouse button, the virtual object 26 is translated in the z axis bythe y component of the differences in mouse locations. When the userholds down the middle mouse button, the virtual object 26 is scaled upor down by a factor of some constant times the y component of thedifference in locations. Typically, the user is performing these mousefunctions with his non-dominant hand (e.g. the left hand for a righthanded user), while using the force reflecting haptic interface in thedominant hand. In another embodiment, the user may hold a second forcereflecting haptic interface, rather than a mouse, in the non-dominanthand to achieve these object manipulations more intuitively.

All of the mouse events are processed in the graphics thread but thevirtual object 26 is transformed in the servo thread. The graphicsthread stores the transformation to be done, and signals the servo loopthat it needs to be done.

Virtual Object Manipulation with the Keyboard

The keyboard manipulation is implemented the same way as for the mouse.Instead of looking at the current and last mouse locations, pressing akey rotates/translates/scales the object by some fixed amount. As withthe mouse, the key events are handled in the graphics thread and thetransformations are carried out in the servo thread.

Import of Surface Objects

In one embodiment, in addition to reading in starter shapes 701represented in the voxel format, the system has the ability to read inobjects represented as polygonal meshes and convert them into GHOST,volumes to be used as initial or a starter shapes 701.

The first part of import is triangularization. The surface must beconverted into a collection of triangles. Currently supported such asSTL (stereolithography) are triangle-based to start with, so the filescan be imported with appropriate readers. For curved objects such asNURBS, a discrete set of approximating triangles have to be generated.

In one embodiment, once the collection of triangles has been defined, itis converted into a 3-D signed distance virtual object 26. This isconceptually similar to the loop rasterizing discussed above, but themore complex primitives (triangles vs. line segments) and need forgreater optimization make the algorithm considerably more involved. Fora detailed discussion, see Bradley Payne and Arthur W. Toga, DistanceField Manipulation of surface Models, IEEE Computer Graphics andApplications, Vol. 12, No. 1, January 1992, pp. 65-71. A brief overviewwill be given here. As in the 2D case, the algorithm works by combininga sign only volume (bitmask) with an undersigned absolute value distanceto obtain a signed distance.

Bitmask Computation

In one embodiment, the bitmask classifies each point in the voxel gridas inside or outside the object. The method does not have to beconcerned about the case of a point lying exactly on the surface, sincethe bitmask just determines the sign of the distance, and +0 and −0 arethe same value. The bitmask is computed slice by slice, where each slicein the volume is a 2-D image. For each of these images, the plane of theimage is intersected with the surface model. Since the surface model isa collection of triangles, each triangle can produce point, linesegment, triangle or the empty set when cut by a plane. The trianglecase only arises when the image plane is coincident with the triangleplane, so the method rules out this degenerate case by perturbing thevertex locations very slightly as needed so none lie exactly on theimage planes. The remaining allowed intersections give a collection ofclosed loops 870 exactly equivalent to the ones used for 2-D distancefield computation described above. The method derives the bitmask fromthe loops 870 the same way as described above.

Distance Computation

In principle, the absolute distance could also be found analogously tothe 2-D case: for each triangle, the method finds the minimum distancefrom the triangle to the voxel location. The method takes the minimum ofall these minima as the absolute minimum distance. In practice, thiswould be slow, so several optimizations are implemented.

Bounding Box Hierarchy

By storing the triangles in a tree of bounding boxes, the method avoidsunnecessary computation. The method refine a list of boxes to includeonly the ones which could potentially include the nearest point, then“open” (that is, inspect the triangles) only for those boxes whichsurvive this initial pruning.

Precomputation

Since the same triangles will be dealt with over and over, the methodprecomputes as much as possible of the information which will be used toget the point-to-triangle distance.

Minimal Computation

In one embodiment, since the 3-D distance field will ultimately bestored as byte values with values of 0 or 255 for all locations asubstantial distance from the isosurface 86, the method can use thebitmask volume to identify only those voxels near enough to the surfaceto have a meaningful distance field value. The method computes onlythose distances, and assigns 0 or 255 directly to values which would beclamped anyway.

Mirroring

Using mirroring, the user can create virtual objects 26 that aresymmetric about a plane. Since most modeled virtual objects 26 are atleast partially symmetrical, this capability is frequently useful. Thestages of mirroring are selecting the mirror plane, editing in mirrorpreview mode (soft mirroring), and executing the mirror operation (hardmirroring), as discussed in more detail below.

Mirror plane selection involves choosing a plane in space and anorientation, where the orientation specifies which side of the plane isconsidered primary and which is the mirror image. Such an oriented planecan be specified as a 4-tuple (a,b,c,d) such that a*x+b*y+c*z+d=0 on theplane, exceeds zero for the primary half-space, and is less than zero inthe mirror image half-space. A plane selection tool is provided. Whenthis tool is active, the user can use the plane selection tool todetermine the orientation of the mirror plane interactively by fitting aplane through the tool position with the normal along the orientation ofthe stylus of the haptic interface device 10. The position of the planeis optionally displayed visually by a grid, and can be inferred from theof the virtual object 26 appearance in a preview mode. After the planeis selected, the corresponding transformation is derived. Thistransformation is the matrix which maps any point in space in theprimary half-space to its mirror image point on the other side of theplane.

Mirror preview mode allows interactive sculpting with the appearance ofreal-time mirroring. In this mode, the virtual object 26 itself isunmodified; only the visual rendering of the virtual object 26 changes.During sculpting operations in this mode, the sculpting tools 28interact with the underlying virtual object 26 in their usual manner.When the scene is drawn, the portion of the virtual object 26 lying inthe primary half-space is shown in the normal (e.g., gray) color.Rendering of this portion is clipped against the mirror plane. Themirror matrix is then used while drawing the virtual object 26 a secondtime in the mirror image half-space, clipping rendering to the otherside of the mirror plane. The object color is also modified (e.g., blue)during this operation. The effect is the appearance of real-time objectmirroring. Each sculpt operation occurring in the primary half-space isvisible at its point of application and again in the reflected object.During mirror previewing, the scene shown is always completelysymmetrical about the mirror plane.

Mirror previewing can be ended by disabling the mirror or by executingthe mirror. If the user disables the mirror, the true underlying virtualobject 26 is shown again and the mirrored area disappears. If the userexecutes the mirror, the virtual object 26 itself is modified to reflectthe desired symmetry.

Hard mirroring is the execution of mirroring inside the virtual object26. Each voxel in the mirror image half-space is replaced by a voxelderived from its reflected point in the primary half-space. Since thereflected point does not in general lie on the voxel grid, the value isderived by trilinear interpolation from the adjacent voxels. Surfaceinformation is then recomputed for the modified portion of the virtualobject 26. After hard mirroring, the plane is inactive and no longerdisplayed (however, its last known position is remembered for subsequentmirroring operations if desired); the virtual object 26 is symmetricalabout the plane unless additional, non-mirrored sculpting is donesubsequently.

Glossary

Constraint

An imposed limitation on the motion of the virtual tool 28.

Density

A scalar property of each single voxel 78, used for defining the shapeof a virtual object 26 in one embodiment.

Density Threshold Value

In one embodiment, the density value which defines the isosurface 86which represents the virtual surface 25. Voxels 78 with densities abovethis value represent interior points on the virtual object 26. Voxels 78with densities below this value represent points outside the virtualobject 26.

Edge

A temporary geometric construct representing a potentially viable movedirection when the virtual tool 28 is contacting the virtual object 26in more than one place.

Gradient

The rate of change of the density values of the voxels 78 along a givenvector in one embodiment.

Haptic Interface Location

The location in the virtual environment which corresponds to the keyposition of the haptic interface device 10 in the real-world space.

Haptic Rendering Process

The process responsible for generating objects 26 in the haptic virtualenvironment and producing high-fidelity force-feedback to the user inreal world space.

Haptic Virtual Environment

A computer representation of a space where a user can interact withvirtual objects 26 through the sense of touch.

Interaction Modes

Settings which alter the interaction between the virtual tool 28 and thevirtual object 26.

Isosurface

A theoretical surface 86 defined by the locus of identical voxel densityvalues.

Panic Escape

A method of extracting the virtual tool 28 from a trapped condition.

Ramp length

In one embodiment, the number of voxels 78 over which density values gofrom their minimum (0) to their maximum (255).

Real World Space

The true three-dimensional physical world in which people live andinteract.

Surface Direction Vector

A vector 101 evaluated at a point in relation to a virtual surface 25.If the point is at the virtual surface 25, this vector 101 is the normalto the surface 25 at that point. If the point is not on the surface 25,the vector 101 represents the approximate shortest distance to thesurface 25 and is also the approximate surface normal at the point wherethe vector 101 would intersect the surface 25.

Tool Surface Contact Point

A location on the virtual surface 25 where the virtual tool 28 is incontact with the virtual surface 25.

Trilinear Interpolation

In one embodiment, a technique for interpolating the densities of nearbyvoxels 78 to derive the density at a location that lies between thediscrete voxel locations.

Virtual Object Modification Process

The process responsible for making changes to the virtual object 26.

Virtual Object

A computer representation of an object.

Virtual Surface

A computer representation of the “skin” of a virtual object 26.

Virtual Tool

A computer representation of a tool which the user uses to interact withthe virtual environment.

Virtual Tool Origin

The location on the virtual tool 28 which strives to be coincident withthe haptic interface location 98.

Voxels

A set of locations in the virtual environment, each storing informationused in defining the shape of a virtual object 26 in one embodiment.

Having described the preferred embodiments of the invention, it will nowbecome apparent to one of skill in the art that other embodimentsincorporating the concepts may be used. It is felt, therefore, that theinvention should not be limited to disclosed embodiments, but rathershould be limited only by the spirit and scope of the following claims.

1-48. (canceled).
 49. A method of modifying a virtual object in avirtual environment, the method comprising the steps of: rendering avirtual object within a virtual environment, the virtual object havingan initial shape; rendering a construction plane within the virtualenvironment; rendering a virtual tool in the virtual environment,wherein the virtual tool is responsive to a user-operated interfacedevice; rendering a sketch on the construction plane in accordance witha movement of the virtual tool; and modifying the initial shape of thevirtual object according to the sketch.
 50. The method of claim 49,wherein the user-operated interface device is a haptic interface device.51. The method of claim 49, wherein the construction plane is capable ofoccupying a smaller cross-sectional area than the virtual object. 52.The method of claim 49, wherein the modifying step comprises at leastone of: (i) adding virtual material to the virtual object; and (ii)removing virtual material from the virtual object.
 53. The method ofclaim 49, wherein the construction plane is capable of being positionedat at least one of: (i) a location external to the virtual object; (ii)a location adjacent to the virtual object; and (iii) a location withinthe virtual object.
 54. A method of modifying a virtual object in avirtual environment, the method comprising the steps of: rendering avirtual object within a virtual environment, the virtual object havingan initial shape; rendering a plurality of construction planes withinthe virtual environment, wherein the construction planes form a stack;rendering a virtual tool in the virtual environment, wherein the virtualtool is responsive to a user-operated interface device; rendering asketch on each of at least two of the construction planes in accordancewith movement of the virtual tool; and modifying the initial shape ofthe virtual object according to an interpolation among the sketches. 55.The method of claim 54, wherein the user-operated interface device is ahaptic interface device.
 56. The method of claim 54, wherein at leastone of the construction planes is capable of occupying a smallercross-sectional area than the virtual object.
 57. The method of claim54, wherein the modifying step comprises at least one of: (i) addingvirtual material to the virtual object; and (ii) removing virtualmaterial from the virtual object.
 58. The method of claim 54, wherein atleast one of the construction planes is capable of being positioned atat least one of: (i) a location external to the virtual object; (ii) alocation adjacent to the virtual object; and (iii) a location within thevirtual object.
 59. An apparatus for modifying a virtual object in avirtual environment, the apparatus comprising: a user-operated interfacedevice; a memory upon which machine-readable code is stored, the codedefining a set of instructions for: rendering a virtual object within avirtual environment, the virtual object having an initial shape;rendering a construction plane within the virtual environment; renderinga virtual tool in the virtual environment, wherein the virtual tool isresponsive to the user-operated haptic interface device; rendering asketch on the construction plane in accordance with a movement of thevirtual tool; and modifying the initial shape of the virtual objectaccording to the sketch.
 60. The apparatus of claim 59, wherein theuser-operated interface device is a haptic interface device.
 61. Theapparatus of claim 59, wherein the virtual object is three-dimensional.